보고서의 목차는 다음과 같다.

1) 개요
  - 데이터셋 정보 
  - 변수 정보
  
2) EDA
  - 기술통계
  - 수치변수의 정규성 검정
    + 통계량과 시각화 정보

3) Model Fitting
  - Linear Regression
  - Logistic Regression
  - LDA
  - QDA
  - Multiple Regression
  - Polynomial model
  - natural spline model
  - Tree

라이브러리 및 데이터셋

library(ggplot2)
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
library(rpart)
library(rpart.plot)
library(dlookr)
## Registered S3 methods overwritten by 'dlookr':
##   method          from  
##   plot.transform  scales
##   print.transform scales
## 
## Attaching package: 'dlookr'
## The following object is masked from 'package:base':
## 
##     transform
library(caret)
## Loading required package: lattice
library(tree)
airline_test_set = read.csv('/Users/ivory/Documents/2024_1_1/static_final_report/dataset/Airline_Passneger_satisfaction/test.csv')
airline_train_set = read.csv('/Users/ivory/Documents/2024_1_1/static_final_report/dataset/Airline_Passneger_satisfaction/train.csv')

# X와 id 변수 제외
airline_test_set = subset(airline_test_set, select = -c(X, id))
airline_train_set = subset(airline_train_set, select = -c(X, id))

개요

데이터셋 정보

해당 데이터셋은 kaggle에 업로드 되어 있는, 항공기 이용 승객들의 만족도 설문조사에 대한 것이다.
이 데이터셋을 이용해서 승객들의 만족도(“Satisfaction”) 예측을 수행할 예정이다.
training set과 test set이 사전에 분리되어 있었기 때문에, 따로 데이터의 분리를 진행하지 않고 제공하고 있는 그래도 사용하기로 한다.
출처 url : https://www.kaggle.com/datasets/teejmahal20/airline-passenger-satisfaction/data

변수정보

head(airline_train_set)
##   Gender     Customer.Type Age  Type.of.Travel    Class Flight.Distance
## 1   Male    Loyal Customer  13 Personal Travel Eco Plus             460
## 2   Male disloyal Customer  25 Business travel Business             235
## 3 Female    Loyal Customer  26 Business travel Business            1142
## 4 Female    Loyal Customer  25 Business travel Business             562
## 5   Male    Loyal Customer  61 Business travel Business             214
## 6 Female    Loyal Customer  26 Personal Travel      Eco            1180
##   Inflight.wifi.service Departure.Arrival.time.convenient
## 1                     3                                 4
## 2                     3                                 2
## 3                     2                                 2
## 4                     2                                 5
## 5                     3                                 3
## 6                     3                                 4
##   Ease.of.Online.booking Gate.location Food.and.drink Online.boarding
## 1                      3             1              5               3
## 2                      3             3              1               3
## 3                      2             2              5               5
## 4                      5             5              2               2
## 5                      3             3              4               5
## 6                      2             1              1               2
##   Seat.comfort Inflight.entertainment On.board.service Leg.room.service
## 1            5                      5                4                3
## 2            1                      1                1                5
## 3            5                      5                4                3
## 4            2                      2                2                5
## 5            5                      3                3                4
## 6            1                      1                3                4
##   Baggage.handling Checkin.service Inflight.service Cleanliness
## 1                4               4                5           5
## 2                3               1                4           1
## 3                4               4                4           5
## 4                3               1                4           2
## 5                4               3                3           3
## 6                4               4                4           1
##   Departure.Delay.in.Minutes Arrival.Delay.in.Minutes            satisfaction
## 1                         25                       18 neutral or dissatisfied
## 2                          1                        6 neutral or dissatisfied
## 3                          0                        0               satisfied
## 4                         11                        9 neutral or dissatisfied
## 5                          0                        0               satisfied
## 6                          0                        0 neutral or dissatisfied
str(airline_train_set)
## 'data.frame':    103904 obs. of  23 variables:
##  $ Gender                           : chr  "Male" "Male" "Female" "Female" ...
##  $ Customer.Type                    : chr  "Loyal Customer" "disloyal Customer" "Loyal Customer" "Loyal Customer" ...
##  $ Age                              : int  13 25 26 25 61 26 47 52 41 20 ...
##  $ Type.of.Travel                   : chr  "Personal Travel" "Business travel" "Business travel" "Business travel" ...
##  $ Class                            : chr  "Eco Plus" "Business" "Business" "Business" ...
##  $ Flight.Distance                  : int  460 235 1142 562 214 1180 1276 2035 853 1061 ...
##  $ Inflight.wifi.service            : int  3 3 2 2 3 3 2 4 1 3 ...
##  $ Departure.Arrival.time.convenient: int  4 2 2 5 3 4 4 3 2 3 ...
##  $ Ease.of.Online.booking           : int  3 3 2 5 3 2 2 4 2 3 ...
##  $ Gate.location                    : int  1 3 2 5 3 1 3 4 2 4 ...
##  $ Food.and.drink                   : int  5 1 5 2 4 1 2 5 4 2 ...
##  $ Online.boarding                  : int  3 3 5 2 5 2 2 5 3 3 ...
##  $ Seat.comfort                     : int  5 1 5 2 5 1 2 5 3 3 ...
##  $ Inflight.entertainment           : int  5 1 5 2 3 1 2 5 1 2 ...
##  $ On.board.service                 : int  4 1 4 2 3 3 3 5 1 2 ...
##  $ Leg.room.service                 : int  3 5 3 5 4 4 3 5 2 3 ...
##  $ Baggage.handling                 : int  4 3 4 3 4 4 4 5 1 4 ...
##  $ Checkin.service                  : int  4 1 4 1 3 4 3 4 4 4 ...
##  $ Inflight.service                 : int  5 4 4 4 3 4 5 5 1 3 ...
##  $ Cleanliness                      : int  5 1 5 2 3 1 2 4 2 2 ...
##  $ Departure.Delay.in.Minutes       : int  25 1 0 11 0 0 9 4 0 0 ...
##  $ Arrival.Delay.in.Minutes         : num  18 6 0 9 0 0 23 0 0 0 ...
##  $ satisfaction                     : chr  "neutral or dissatisfied" "neutral or dissatisfied" "satisfied" "neutral or dissatisfied" ...

observations of training set : 103904
number of variables : 23

  • “Gender” : 승객의 성별 (Female, Male)
  • “Customer.Type” : 고객 충성도 유형 (Loyal Customer, disLoyal Customer)
  • “Age” : 승객의 나이
  • “Type.of.Travel” : 승객의 비행 목적 (Business travel, Personal Travel)
  • “Class” : 승객의 비행기 좌석 등급 (Business, Eco, Ohter)
  • “Flight.Distance” : 승객별 비행 거리
  • “Inflight.wifi.service” : 기내 와이파이 서비스 만족도 (0: 해당 없음; 1-5)
  • “Departure.Arrival.time.convenient” : 출발/도착 시간 편의성 만족도
  • “Ease.of.Online.booking” : 온라인 예약 용이성 만족도(0: 해당 없음; 1-5)
  • “Gate.location” : 게이트 위치 만족도(0: 해당 없음; 1-5)
  • “Food.and.drink” : 음식 및 음료 만족도(0: 해당 없음; 1-5)
  • “Online.boarding” : 온라인 탑승 수속 만족도(0: 해당 없음; 1-5)
  • “Seat.comfort” : 좌석 편안함 만족도(0: 해당 없음; 1-5)
  • “Inflight.entertainment” : 기내 오락 만족도(0: 해당 없음; 1-5)
  • “On.board.service” : 기내 서비스 만족도(0: 해당 없음; 1-5)
  • “Leg.room.service” : 다리 공간 서비스 만족도(0: 해당 없음; 1-5)
  • “Baggage.handling” : 수하물 처리 만족도(0: 해당 없음; 1-5)
  • “Checkin.service” : 체크인 서비스 만족도(0: 해당 없음; 1-5)
  • “Inflight.service” : 기내 서비스 만족도(0: 해당 없음; 1-5)
  • “Cleanliness” : 청결도 만족도(0: 해당 없음; 1-5)
  • “Departure.Delay.in.Minutes” : 출발 지연 시간 (분 단위)
  • “Arrival.Delay.in.Minutes” : 도착 지연 시간 (분 단위)
  • “satisfaction” : 항공사 만족도 (만족, 보통 또는 불만족)

X와 id 변수는 satisfaction을 예측하는데 유의미하지 않으므로 제외한다.

numerical 변수 : [“Age”, “Flight.Distance”, “Departure.Delay.in.Minutes”, “Arrival.Delay.in.Minutes”] categorical 변수 : [“Gender”, “Customer.Type”, “Type.of.Travel”, “Class”, “Inflight.wifi.service”, “Departure.Arrival.time.convenient”, “Ease.of.Online.booking”, “Gate.location”, “Food.and.drink”, “Online.boarding”, “Seat.comfort”, “Inflight.entertainment”, “On.board.service”, “Leg.room.service”, “Baggage.handing”, “Checkin.service”, “Inflight.service”, “Cleanliness”, “satisfaction”]

unique_values = unique(airline_train_set$satisfaction)
unique_values
## [1] "neutral or dissatisfied" "satisfied"

satisfaction는 “satisfied” 또는 “neutral or dissatisfied”가 존재한다. 해당 명목형 변수를 이진 변수로 변환한다.

airline_test_set$satisfaction = ifelse(airline_test_set$satisfaction == "satisfied", 1, 0)
airline_train_set$satisfaction = ifelse(airline_train_set$satisfaction == "satisfied", 1, 0)
table(airline_train_set$satisfaction)
## 
##     0     1 
## 58879 45025


total : 103904 satisfied : 45025
neutral or dissatisfied : 58879

“satisfied” : “neutral or dissatisfied” = 4.3 : 5.7

table(airline_test_set$satisfaction)
## 
##     0     1 
## 14573 11403


total : 25976 satisfied : 11403
neutral or dissatisfied : 14573

“satisfied” : “neutral or dissatisfied” = 4.4 : 5.6

conclustion

training set 과 test set의 비율은 8 대 2이다. 각 data set에서 satisfaction 변수의 클래스 비율은 약 4.4 대 5.6정도로 큰 차이가 나지 않는다.

EDA

summary(airline_train_set)
##     Gender          Customer.Type           Age        Type.of.Travel    
##  Length:103904      Length:103904      Min.   : 7.00   Length:103904     
##  Class :character   Class :character   1st Qu.:27.00   Class :character  
##  Mode  :character   Mode  :character   Median :40.00   Mode  :character  
##                                        Mean   :39.38                     
##                                        3rd Qu.:51.00                     
##                                        Max.   :85.00                     
##                                                                          
##     Class           Flight.Distance Inflight.wifi.service
##  Length:103904      Min.   :  31    Min.   :0.00         
##  Class :character   1st Qu.: 414    1st Qu.:2.00         
##  Mode  :character   Median : 843    Median :3.00         
##                     Mean   :1189    Mean   :2.73         
##                     3rd Qu.:1743    3rd Qu.:4.00         
##                     Max.   :4983    Max.   :5.00         
##                                                          
##  Departure.Arrival.time.convenient Ease.of.Online.booking Gate.location  
##  Min.   :0.00                      Min.   :0.000          Min.   :0.000  
##  1st Qu.:2.00                      1st Qu.:2.000          1st Qu.:2.000  
##  Median :3.00                      Median :3.000          Median :3.000  
##  Mean   :3.06                      Mean   :2.757          Mean   :2.977  
##  3rd Qu.:4.00                      3rd Qu.:4.000          3rd Qu.:4.000  
##  Max.   :5.00                      Max.   :5.000          Max.   :5.000  
##                                                                          
##  Food.and.drink  Online.boarding  Seat.comfort   Inflight.entertainment
##  Min.   :0.000   Min.   :0.00    Min.   :0.000   Min.   :0.000         
##  1st Qu.:2.000   1st Qu.:2.00    1st Qu.:2.000   1st Qu.:2.000         
##  Median :3.000   Median :3.00    Median :4.000   Median :4.000         
##  Mean   :3.202   Mean   :3.25    Mean   :3.439   Mean   :3.358         
##  3rd Qu.:4.000   3rd Qu.:4.00    3rd Qu.:5.000   3rd Qu.:4.000         
##  Max.   :5.000   Max.   :5.00    Max.   :5.000   Max.   :5.000         
##                                                                        
##  On.board.service Leg.room.service Baggage.handling Checkin.service
##  Min.   :0.000    Min.   :0.000    Min.   :1.000    Min.   :0.000  
##  1st Qu.:2.000    1st Qu.:2.000    1st Qu.:3.000    1st Qu.:3.000  
##  Median :4.000    Median :4.000    Median :4.000    Median :3.000  
##  Mean   :3.382    Mean   :3.351    Mean   :3.632    Mean   :3.304  
##  3rd Qu.:4.000    3rd Qu.:4.000    3rd Qu.:5.000    3rd Qu.:4.000  
##  Max.   :5.000    Max.   :5.000    Max.   :5.000    Max.   :5.000  
##                                                                    
##  Inflight.service  Cleanliness    Departure.Delay.in.Minutes
##  Min.   :0.00     Min.   :0.000   Min.   :   0.00           
##  1st Qu.:3.00     1st Qu.:2.000   1st Qu.:   0.00           
##  Median :4.00     Median :3.000   Median :   0.00           
##  Mean   :3.64     Mean   :3.286   Mean   :  14.82           
##  3rd Qu.:5.00     3rd Qu.:4.000   3rd Qu.:  12.00           
##  Max.   :5.00     Max.   :5.000   Max.   :1592.00           
##                                                             
##  Arrival.Delay.in.Minutes  satisfaction   
##  Min.   :   0.00          Min.   :0.0000  
##  1st Qu.:   0.00          1st Qu.:0.0000  
##  Median :   0.00          Median :0.0000  
##  Mean   :  15.18          Mean   :0.4333  
##  3rd Qu.:  13.00          3rd Qu.:1.0000  
##  Max.   :1584.00          Max.   :1.0000  
##  NA's   :310

[Gender, Customer.Type, Type.of.Travel, satisfaction] 변수를 제외한 나머지는 모두 수치변수에 해당한다.

“Satisfaction”을 반응변수로 선택한다.

#training set에 대한 결측치 확인
sum(is.na(airline_train_set))
## [1] 310
colSums(is.na(airline_train_set))
##                            Gender                     Customer.Type 
##                                 0                                 0 
##                               Age                    Type.of.Travel 
##                                 0                                 0 
##                             Class                   Flight.Distance 
##                                 0                                 0 
##             Inflight.wifi.service Departure.Arrival.time.convenient 
##                                 0                                 0 
##            Ease.of.Online.booking                     Gate.location 
##                                 0                                 0 
##                    Food.and.drink                   Online.boarding 
##                                 0                                 0 
##                      Seat.comfort            Inflight.entertainment 
##                                 0                                 0 
##                  On.board.service                  Leg.room.service 
##                                 0                                 0 
##                  Baggage.handling                   Checkin.service 
##                                 0                                 0 
##                  Inflight.service                       Cleanliness 
##                                 0                                 0 
##        Departure.Delay.in.Minutes          Arrival.Delay.in.Minutes 
##                                 0                               310 
##                      satisfaction 
##                                 0

training set의 Arrival.Delay.in.Minutes 변수에 310개의 결측변수가 존재함을 확인했다. 해당 결측치를 제거하기로 한다.

#test set에 대한 결측치 확인
sum(is.na(airline_test_set))
## [1] 83
colSums(is.na(airline_test_set))
##                            Gender                     Customer.Type 
##                                 0                                 0 
##                               Age                    Type.of.Travel 
##                                 0                                 0 
##                             Class                   Flight.Distance 
##                                 0                                 0 
##             Inflight.wifi.service Departure.Arrival.time.convenient 
##                                 0                                 0 
##            Ease.of.Online.booking                     Gate.location 
##                                 0                                 0 
##                    Food.and.drink                   Online.boarding 
##                                 0                                 0 
##                      Seat.comfort            Inflight.entertainment 
##                                 0                                 0 
##                  On.board.service                  Leg.room.service 
##                                 0                                 0 
##                  Baggage.handling                   Checkin.service 
##                                 0                                 0 
##                  Inflight.service                       Cleanliness 
##                                 0                                 0 
##        Departure.Delay.in.Minutes          Arrival.Delay.in.Minutes 
##                                 0                                83 
##                      satisfaction 
##                                 0

Arrival.Delay.in.Minutes에 83개의 결측변수가 존재함을 확인했다. 해당 결측치를 제거하기로 한다.

#training set의 결측치 제거
airline_train_set = na.omit(airline_train_set)
#test set의 결측치 제거
airline_test_set = na.omit(airline_test_set)
결측치 제거 확인
#training set에 대한 결측치 확인
sum(is.na(airline_train_set))
## [1] 0
colSums(is.na(airline_train_set))
##                            Gender                     Customer.Type 
##                                 0                                 0 
##                               Age                    Type.of.Travel 
##                                 0                                 0 
##                             Class                   Flight.Distance 
##                                 0                                 0 
##             Inflight.wifi.service Departure.Arrival.time.convenient 
##                                 0                                 0 
##            Ease.of.Online.booking                     Gate.location 
##                                 0                                 0 
##                    Food.and.drink                   Online.boarding 
##                                 0                                 0 
##                      Seat.comfort            Inflight.entertainment 
##                                 0                                 0 
##                  On.board.service                  Leg.room.service 
##                                 0                                 0 
##                  Baggage.handling                   Checkin.service 
##                                 0                                 0 
##                  Inflight.service                       Cleanliness 
##                                 0                                 0 
##        Departure.Delay.in.Minutes          Arrival.Delay.in.Minutes 
##                                 0                                 0 
##                      satisfaction 
##                                 0
#test set에 대한 결측치 확인
sum(is.na(airline_test_set))
## [1] 0
colSums(is.na(airline_test_set))
##                            Gender                     Customer.Type 
##                                 0                                 0 
##                               Age                    Type.of.Travel 
##                                 0                                 0 
##                             Class                   Flight.Distance 
##                                 0                                 0 
##             Inflight.wifi.service Departure.Arrival.time.convenient 
##                                 0                                 0 
##            Ease.of.Online.booking                     Gate.location 
##                                 0                                 0 
##                    Food.and.drink                   Online.boarding 
##                                 0                                 0 
##                      Seat.comfort            Inflight.entertainment 
##                                 0                                 0 
##                  On.board.service                  Leg.room.service 
##                                 0                                 0 
##                  Baggage.handling                   Checkin.service 
##                                 0                                 0 
##                  Inflight.service                       Cleanliness 
##                                 0                                 0 
##        Departure.Delay.in.Minutes          Arrival.Delay.in.Minutes 
##                                 0                                 0 
##                      satisfaction 
##                                 0
plot_outlier(airline_train_set)


[“Age”, “Inflight.wifi.service”, “Online.boarding”, “Gate.location”]


[“Flight.Distance”]


[“Online.boarding”, “Seat.comfort”, “Inflight.entertainment”, “Baggage.handling”, “Cleanliness”, “On.board.service”, “Leg.room.fservice”, “Inflight.service”]


[“Flight.Distance”, “Departure.Delay.in.Minutes”, “Arrival.Delay.in.Minutes”]

위의 변수들 중에서 [“Age”, “Inflight.wifi.service”, “Online.boarding”, “Gate.location”] 4가지의 변수만 정규분포를 따르고 있었다.

이 중에서 Age만 numerical 변수이고 나머지는 categorical 변수이다.

plot_normality(airline_train_set)

Aae의 경우, 완벽한 정규분포라고는 할 수 없지만, Q-Q plot을 통해서 어느정도 정규성을 따른다는 것을 확인했다.

변수들 중에서 [“Flight.Distance”, “Departure.Delay.in.Minutes”, “Arrival.Delay.in.Minutes”]는 log transformation 한 경우에 정규 분포의 형태를 띄므로 해당 변수들에 대해서 log transformation을 진행하도록 한다.

airline_test_set$Flight.Distance_log = log(airline_test_set$Flight.Distance + 1)
airline_test_set$Departure.Delay.in.Minutes_log = log(airline_test_set$Departure.Delay.in.Minutes + 1)
airline_test_set$Arrival.Delay.in.Minutes_log = log(airline_test_set$Arrival.Delay.in.Minutes + 1)
airline_train_set$Flight.Distance_log = log(airline_train_set$Flight.Distance + 1)
airline_train_set$Departure.Delay.in.Minutes_log = log(airline_train_set$Departure.Delay.in.Minutes + 1)
airline_train_set$Arrival.Delay.in.Minutes_log = log(airline_train_set$Arrival.Delay.in.Minutes + 1)

Model Fitting

Linear Regression model

원래의 목적은 satisfaction 변수를 추정하는 것이지만, 해당 변수는 regression모델에는 적합하지 않다. 따라서, 기내 서비스 몬족도를 예측하는(Inflight.service) 모델을 만드는 것을 목표로 한다.

#log scale로 변환하기 전 변수들 제외
inflight_service_lm = lm(Inflight.service ~ . -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)
summary(inflight_service_lm)
## 
## Call:
## lm(formula = Inflight.service ~ . - Flight.Distance - Departure.Delay.in.Minutes - 
##     Arrival.Delay.in.Minutes, data = airline_train_set)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.9642 -0.3341  0.0092  0.4125  3.4174 
## 
## Coefficients:
##                                     Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                        1.408e+00  2.744e-02  51.322  < 2e-16 ***
## GenderMale                         4.548e-02  5.097e-03   8.921  < 2e-16 ***
## Customer.TypeLoyal Customer       -2.409e-01  8.480e-03 -28.409  < 2e-16 ***
## Age                               -3.407e-03  1.807e-04 -18.851  < 2e-16 ***
## Type.of.TravelPersonal Travel      2.352e-01  8.396e-03  28.011  < 2e-16 ***
## ClassEco                          -1.435e-01  7.435e-03 -19.303  < 2e-16 ***
## ClassEco Plus                     -1.170e-01  1.118e-02 -10.469  < 2e-16 ***
## Inflight.wifi.service             -1.477e-02  3.040e-03  -4.860 1.18e-06 ***
## Departure.Arrival.time.convenient  3.215e-02  2.148e-03  14.967  < 2e-16 ***
## Ease.of.Online.booking            -6.975e-03  2.999e-03  -2.325   0.0201 *  
## Gate.location                      1.004e-03  2.432e-03   0.413   0.6798    
## Food.and.drink                    -8.296e-02  2.799e-03 -29.645  < 2e-16 ***
## Online.boarding                   -6.166e-04  2.728e-03  -0.226   0.8212    
## Seat.comfort                      -7.032e-02  2.962e-03 -23.740  < 2e-16 ***
## Inflight.entertainment             3.126e-01  3.625e-03  86.222  < 2e-16 ***
## On.board.service                   1.754e-01  2.576e-03  68.104  < 2e-16 ***
## Leg.room.service                   6.528e-02  2.214e-03  29.484  < 2e-16 ***
## Baggage.handling                   3.461e-01  2.765e-03 125.198  < 2e-16 ***
## Checkin.service                    7.510e-02  2.225e-03  33.760  < 2e-16 ***
## Cleanliness                       -1.089e-01  3.262e-03 -33.383  < 2e-16 ***
## satisfaction                       8.742e-02  7.669e-03  11.399  < 2e-16 ***
## Flight.Distance_log               -2.662e-03  3.106e-03  -0.857   0.3914    
## Departure.Delay.in.Minutes_log     3.679e-05  2.666e-03   0.014   0.9890    
## Arrival.Delay.in.Minutes_log      -1.626e-02  2.655e-03  -6.125 9.09e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.816 on 103570 degrees of freedom
## Multiple R-squared:  0.5184, Adjusted R-squared:  0.5182 
## F-statistic:  4846 on 23 and 103570 DF,  p-value: < 2.2e-16

모든 변수를 이용해서 모델을 생성한 결과,
[“Gate.location”, “Online.boarding”, “Flight.Distance_log”, “Departure.Delay.in.Minutes_log”]는 Inflight.service 변수 예측에 통계적으로 유의미하지 않음을 확인했다.
따라서, 해당 변수를 제외하고 새로운 모델을 만들기로 한다.

#log scale로 변환하기 전 변수들 제외
inflight_service_lm = lm(Inflight.service ~ . -Gate.location -Online.boarding -Flight.Distance_log -Departure.Delay.in.Minutes_log -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)
summary(inflight_service_lm)
## 
## Call:
## lm(formula = Inflight.service ~ . - Gate.location - Online.boarding - 
##     Flight.Distance_log - Departure.Delay.in.Minutes_log - Flight.Distance - 
##     Departure.Delay.in.Minutes - Arrival.Delay.in.Minutes, data = airline_train_set)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.9650 -0.3341  0.0090  0.4125  3.4214 
## 
## Coefficients:
##                                    Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                        1.392267   0.017337  80.309  < 2e-16 ***
## GenderMale                         0.045562   0.005086   8.958  < 2e-16 ***
## Customer.TypeLoyal Customer       -0.242568   0.008304 -29.211  < 2e-16 ***
## Age                               -0.003404   0.000179 -19.024  < 2e-16 ***
## Type.of.TravelPersonal Travel      0.235432   0.008353  28.187  < 2e-16 ***
## ClassEco                          -0.141809   0.007183 -19.743  < 2e-16 ***
## ClassEco Plus                     -0.115070   0.010952 -10.506  < 2e-16 ***
## Inflight.wifi.service             -0.014794   0.002949  -5.017 5.25e-07 ***
## Departure.Arrival.time.convenient  0.032506   0.002025  16.054  < 2e-16 ***
## Ease.of.Online.booking            -0.006954   0.002830  -2.457    0.014 *  
## Food.and.drink                    -0.082972   0.002797 -29.668  < 2e-16 ***
## Seat.comfort                      -0.070524   0.002882 -24.473  < 2e-16 ***
## Inflight.entertainment             0.312758   0.003604  86.783  < 2e-16 ***
## On.board.service                   0.175355   0.002572  68.178  < 2e-16 ***
## Leg.room.service                   0.065230   0.002205  29.587  < 2e-16 ***
## Baggage.handling                   0.346136   0.002765 125.207  < 2e-16 ***
## Checkin.service                    0.074993   0.002214  33.879  < 2e-16 ***
## Cleanliness                       -0.109026   0.003243 -33.625  < 2e-16 ***
## satisfaction                       0.086893   0.007471  11.631  < 2e-16 ***
## Arrival.Delay.in.Minutes_log      -0.016246   0.001567 -10.365  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.816 on 103574 degrees of freedom
## Multiple R-squared:  0.5183, Adjusted R-squared:  0.5183 
## F-statistic:  5867 on 19 and 103574 DF,  p-value: < 2.2e-16

해당 모델의 설명력(R-squared)는 0.5183임을 확인할 수 있었다.

correlation
library(corrplot)
## corrplot 0.92 loaded
selected_data = select(airline_train_set, Inflight.service, Age, Inflight.wifi.service, Departure.Arrival.time.convenient, Ease.of.Online.booking, Food.and.drink, Seat.comfort, Inflight.entertainment, On.board.service, Leg.room.service, Baggage.handling, Checkin.service, Cleanliness)

selected_data = rename(selected_data, Inflight = Inflight.service,
                       wifi = Inflight.wifi.service,
                       depart.a = Departure.Arrival.time.convenient,
                       book = Ease.of.Online.booking,
                       food = Food.and.drink,
                       seat = Seat.comfort,
                       enter = Inflight.entertainment,
                       board = On.board.service,
                       leg = Leg.room.service,
                       bagg = Baggage.handling,
                       checkin = Checkin.service,
                       clean = Cleanliness)
correlation_matrix = cor(selected_data, use="complete.obs")
corrplot(correlation_matrix, method = "number")

Inflight.service와 각각의 예측변수 사이의 상관관계 확인 후, 0.4 이상의 값을 갖는 변수들만을 선택해서 linear regression 모델의 예측변수로 선정하기로 한다.
이때, 선택된 예측변수는 [“Inflight.entertainment”, “On.board.service”, “Baggage.handling”] 이다.

inflight_service_final_lm = lm(Inflight.service ~ Inflight.entertainment + On.board.service + Baggage.handling, data = airline_train_set)
summary(inflight_service_final_lm)
## 
## Call:
## lm(formula = Inflight.service ~ Inflight.entertainment + On.board.service + 
##     Baggage.handling, data = airline_train_set)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.8204 -0.4340 -0.0249  0.4152  3.3615 
## 
## Coefficients:
##                        Estimate Std. Error t value Pr(>|t|)    
## (Intercept)            0.842966   0.009628   87.55   <2e-16 ***
## Inflight.entertainment 0.110287   0.002233   49.40   <2e-16 ***
## On.board.service       0.245065   0.002502   97.94   <2e-16 ***
## Baggage.handling       0.440141   0.002675  164.51   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.8504 on 103590 degrees of freedom
## Multiple R-squared:  0.4767, Adjusted R-squared:  0.4767 
## F-statistic: 3.145e+04 on 3 and 103590 DF,  p-value: < 2.2e-16

상관관계를 기반으로 예측변수를 선정했지만, 오히려 R-squared 값이 하락했다.
R-squared 변화 : 0.5183 -> 0.4767(-0.0416)
설명력이 약간 하락했지만, 그 차이가 미미하고 사용한 예측변수의 수도 적기 때문에 해당 모델을 최종 모델로 선정한다.

prediction
predictions = predict(inflight_service_final_lm, newdata = airline_test_set)
predictions[1:5]
##        1        2        3        4        5 
## 4.820431 4.024938 3.364223 1.638459 2.433952

소수 첫쨰 자리에서 반올림한다.

predicted = round(predictions, 0)
actual = airline_test_set$Inflight.service
length(predicted)
## [1] 25893
length(actual)
## [1] 25893
Evaluation
predicted = factor(predicted)
actual = factor(actual)

conf_matrix = confusionMatrix(predicted, actual)
## Warning in levels(reference) != levels(data): longer object length is not a
## multiple of shorter object length
## Warning in confusionMatrix.default(predicted, actual): Levels are not in the
## same order for reference and data. Refactoring data to match.
print(conf_matrix)
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    0    1    2    3    4    5
##          0    0    0    0    0    0    0
##          1    0    0    0    0    0    0
##          2    2  915 1612  266  144  140
##          3    0  495  827 3586 1977  545
##          4    0  344  359 1092 6803 2917
##          5    0   21   40   61  428 3319
## 
## Overall Statistics
##                                           
##                Accuracy : 0.5917          
##                  95% CI : (0.5856, 0.5977)
##     No Information Rate : 0.3612          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.4414          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                       Class: 0 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity          0.000e+00  0.00000  0.56801   0.7165   0.7274   0.4796
## Specificity          1.000e+00  1.00000  0.93637   0.8160   0.7151   0.9710
## Pos Pred Value             NaN      NaN  0.52355   0.4826   0.5908   0.8578
## Neg Pred Value       9.999e-01  0.93145  0.94626   0.9231   0.8227   0.8365
## Prevalence           7.724e-05  0.06855  0.10960   0.1933   0.3612   0.2673
## Detection Rate       0.000e+00  0.00000  0.06226   0.1385   0.2627   0.1282
## Detection Prevalence 0.000e+00  0.00000  0.11891   0.2870   0.4447   0.1494
## Balanced Accuracy    5.000e-01  0.50000  0.75219   0.7662   0.7213   0.7253

Accuracy : 0.59

Class 0

민감도가 0으로 클래스가 0인 경우에 하나도 올바르게 감지하지 못했다는 것을 의미한다.
특이도는 1으로 크래스가 0이 아닌 경우에 완벽하게 감지했다는 것을 의미한다.
이러한 결과가 나온 이유는 class가 0인 레이블이 거의 존재하지 않기 때문이다.

Class 1

민감도는 0, 특이도는 1으로 class가 1인 경우 1로 제대로 예측을 하지 못 했다는 것을 확인할 수 있다.

Class 2

민감도가 약 0.57으로 클래스 2 예측 성능이 상대적으로 나은 것을 알 수 있다.
특이도도는 약 0.94으로 클래스 2가 아닌 것을 잘 구분하는 것을 확인할 수 있다.
balanced accuacy는 0.75으로 이 클래스에 대한 모델의 성능이 상대적으로 괜찮은 것을 알 수 있다.

Class 3

민감도가 0.72로 클래스의 예측 능력이 이전의 클래스보다 높은 것을 알 수 있다. 특이도는 0.82이다.

Class 4

민감도가 0.73으로 해당 모델이 class 4를 예측하는 데 가장 높은 성능을 보인다.

Class 5

민감도는 0.48, 특이도는0.97이다.

conclusion : Linear regression

해당 모델은 클래스(0,1,2)에서 낮은 성능을 보이지만, 다른 클래스(3,4,5)에서는 낭느 결과를 보이고 있는 것을 확인했다. 해당 결과를 보인 이유는, Inflight.service의 분포가 left-skewed distribution을 따르기 때문이라고 결론을 내린다.

Logistic regresion

해당 파트에서는 Logistic model을 이용하여 항공기 이용객들의 satisfaction을 예측하는 것을 목표로 한다.

일반적인 상식으로 생각해본다면, satisfaction과 가장 관련이 큰 요소들은 [Customer.Type, Departure.Arrival.time.convenient, On.board.service, Seat.comfort, Cleanliness] 이다. 해당 변수들을 중점적으로 종속변수를 예측할 것이다.

table(airline_train_set$satisfaction)
## 
##     0     1 
## 58697 44897

<obs : 103594>
“보통 또는 불만족” : 58697(0.57) “만족” : 44897(0.43)

만족으로 답한 승객보다 보통 또는 불만족이라고 답한 승객의 수가 많은 것을 확인했다.

satisfcation ~ Customer.Type
CrossTable = table(airline_train_set$satisfaction, airline_train_set$Customer.Type)
mosaicplot(CrossTable, main="Mosaic Plot")

만족이라고 답한 승객 중 대부분은 충성도가 높은 고객이었다.
하지만, 보통 또는 불만족인 경우에도 충성도가 높은 고객의 비율이 높은 것을 확인할 수 있다.

ggplot(airline_train_set, aes(x=factor(Customer.Type), fill=factor(satisfaction))) +
  geom_bar(position="fill") +
  labs(x="Category Variable", y="Proportion", fill="Target")

예측변수가 특정 클래스로 편향되어 있기 때문에, 비율을 기준으로 확인했다. 해당 항공사 데이터의 경우, 충성 고객임에도 만족하는 경우가 50% 미만인 것을 알 수 있다. 충성고객이어야 그나마 만족이라고 답변을 한다는 것을 알 수 있다.

satisfcation ~ Departure.Arrival.time.convenient
ggplot(airline_train_set, aes(x=factor(Departure.Arrival.time.convenient), fill=factor(satisfaction))) +
  geom_bar(position="fill") +
  labs(x="time.convenient", y="Proportion", title="Proportion")

satisfaction을 예측하는 데, time.convenuent의 클래스별 유의미한 의미가 보이지 않는다. 해당 변수를 제외하기로 한다.

satisfcation ~ On.board.service
ggplot(airline_train_set, aes(x=factor(On.board.service), fill=factor(satisfaction))) +
  geom_bar(position="fill") +
  labs(x="On.board.service", y="Proportion", title="Proportion")

On.baord.service(기내 서비스 만족도)가 높을수록 satisfaction에 만족이라고 답한 비율이 올라가는 것을 확인할 수 있다. 이것으로 On.board.service와 satisfaction은 유의미한 관계를 갖다는 것을 알 수 있다.

satisfcation ~ Seat.comfort
ggplot(airline_train_set, aes(x=factor(Seat.comfort), fill=factor(satisfaction))) +
  geom_bar(position="fill") +
  labs(x="Seat.comfort", y="Proportion", title="Proportion")

Seat.comfort의 평가가 높아질수록 satisfaction의 비율이 높아지는 것을 확인할 수 있다.

satisfcation ~ Cleanliness
ggplot(airline_train_set, aes(x=factor(Cleanliness), fill=factor(satisfaction))) +
  geom_bar(position="fill") +
  labs(x="Cleanliness", y="Proportion", title="Proportion")

satisifaction와 Cleanliness 변수의 관계역시 양적 상관관계를 갖는 것을 확인할 수 있다.

결론적으로, satisfaction을 예측하는 예측변수로 [“Customer.Type”, “On.board.service”, “Seat.comfort”, “Cleanliness”]를 사용하기로 한다.

model fitting : logistic regression
logistic_model = glm(satisfaction ~ Customer.Type + On.board.service + Seat.comfort + Cleanliness, data = airline_train_set, family = "binomial")
summary(logistic_model)
## 
## Call:
## glm(formula = satisfaction ~ Customer.Type + On.board.service + 
##     Seat.comfort + Cleanliness, family = "binomial", data = airline_train_set)
## 
## Coefficients:
##                              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)                 -5.092707   0.037064 -137.40   <2e-16 ***
## Customer.TypeLoyal Customer  0.919842   0.020282   45.35   <2e-16 ***
## On.board.service             0.532996   0.005968   89.30   <2e-16 ***
## Seat.comfort                 0.408277   0.007442   54.86   <2e-16 ***
## Cleanliness                  0.229151   0.007316   31.32   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 141768  on 103593  degrees of freedom
## Residual deviance: 115889  on 103589  degrees of freedom
## AIC: 115899
## 
## Number of Fisher Scoring iterations: 4

모든 예측변수들이 통계적으로 유의미함을 보인다.

Evaluation
probabilities = predict(logistic_model, newdata = airline_test_set, type = "response")
predicted_classes <- ifelse(probabilities > 0.5, 1, 0)

conf_matrix = confusionMatrix(factor(predicted_classes), factor(airline_test_set$satisfaction))
print(conf_matrix)
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction     0     1
##          0 11544  3926
##          1  2984  7439
##                                           
##                Accuracy : 0.7331          
##                  95% CI : (0.7277, 0.7385)
##     No Information Rate : 0.5611          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.4532          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##             Sensitivity : 0.7946          
##             Specificity : 0.6546          
##          Pos Pred Value : 0.7462          
##          Neg Pred Value : 0.7137          
##              Prevalence : 0.5611          
##          Detection Rate : 0.4458          
##    Detection Prevalence : 0.5975          
##       Balanced Accuracy : 0.7246          
##                                           
##        'Positive' Class : 0               
## 
tp = 7439
fp = 2984
fn = 3926
tn = 11544

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * (precision*recall / (precision+recall))

cat("Precision:", precision, "\n")
## Precision: 0.7137101
cat("Recall:", recall, "\n")
## Recall: 0.6545535
cat("F1 Score:", f1, "\n")
## F1 Score: 0.6828529

f1-score가 0.68으로, 해당 로지스틱 모델의 성능이 좋게 나타나진 않았다.

LDA

LDA는 서로 다른 카테고리 사이의 거리를 최대화하는 차원축소의 기법이다. 보통은 분류 카테고리가 3개 이상일 때 사용하고, 각각의 클래스가 정규분포를 따른다고 가정을 하는데 해당 데이터셋에는 적합하지가 않다. LDA의 가정을 만족하지 않으므로 대신 QDA를 진행하도록 한다.

QDA

library(MASS)
## 
## Attaching package: 'MASS'
## The following object is masked from 'package:dplyr':
## 
##     select
qda_model = qda(satisfaction ~ Customer.Type + On.board.service + Seat.comfort + Cleanliness, data = airline_train_set)
summary(qda_model)
##         Length Class  Mode     
## prior    2     -none- numeric  
## counts   2     -none- numeric  
## means    8     -none- numeric  
## scaling 32     -none- numeric  
## ldet     2     -none- numeric  
## lev      2     -none- character
## N        1     -none- numeric  
## call     3     -none- call     
## terms    3     terms  call     
## xlevels  1     -none- list
# 예측 및 모델 평가
probabilities_qda = predict(qda_model, newdata = airline_test_set)
confusionMatrix(probabilities_qda$class, factor(airline_test_set$satisfaction))
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction     0     1
##          0 10452  3137
##          1  4076  8228
##                                           
##                Accuracy : 0.7214          
##                  95% CI : (0.7159, 0.7269)
##     No Information Rate : 0.5611          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.4395          
##                                           
##  Mcnemar's Test P-Value : < 2.2e-16       
##                                           
##             Sensitivity : 0.7194          
##             Specificity : 0.7240          
##          Pos Pred Value : 0.7692          
##          Neg Pred Value : 0.6687          
##              Prevalence : 0.5611          
##          Detection Rate : 0.4037          
##    Detection Prevalence : 0.5248          
##       Balanced Accuracy : 0.7217          
##                                           
##        'Positive' Class : 0               
## 
tp = 8228
fp = 4076
fn = 3137
tn = 10452

precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * (precision*recall / (precision+recall))

cat("Precision:", precision, "\n")
## Precision: 0.6687256
cat("Recall:", recall, "\n")
## Recall: 0.7239771
cat("F1 Score:", f1, "\n")
## F1 Score: 0.6952554

logistic과 QDA를 비교했을 때, QDA의 경우가 precision이 낮아지고, recall이 높아져서 근소하게 f1-score값이 logistic model 보다는 높게 나타났다. 근러나 그 차이가 근소하기 때문에 유의미한 차이가 있다고는 말하기 어렵다.

multiple regression

satisfaction model에는 수치형 변수를 종속 변수로 설정하는 것이 이상적이지만, 해당 데이터셋의 대부분의 변수의 종류가 categorical 이므로, 그 중에서 Ordinal 변수를 예측하는 것으로 한다.
그 중에서 Inflight.service를 예측하는 것을 목표로 한다.

EDA

우선, Inflight.service와 다른 변수들 간의 상관관계를 확인한다.

library(dplyr)
# 1. correlation
num_vars <- airline_train_set %>% dplyr::select(Inflight.service, Customer.Type, Type.of.Travel, Class, Flight.Distance_log, Inflight.wifi.service, Seat.comfort, Inflight.entertainment, On.board.service, Cleanliness)
#one-hot encoding

## Customer.Type
Customer.type_factor = factor(num_vars$Customer.Type, levels = unique(num_vars$Customer.Type))
num_vars$Customer.Type = as.integer(Customer.type_factor)

## Type.of.Travel
Type.of.travel_factor = factor(num_vars$Type.of.Travel, levels = unique(num_vars$Type.of.Travel))
num_vars$Type.of.Travel = as.integer(Type.of.travel_factor)

## Class
Class_factor = factor(num_vars$Class, levels = unique(num_vars$Class))
num_vars$Class = as.integer(Class_factor)
str(num_vars)
## 'data.frame':    103594 obs. of  10 variables:
##  $ Inflight.service      : int  5 4 4 4 3 4 5 5 1 3 ...
##  $ Customer.Type         : int  1 2 1 1 1 1 1 1 1 2 ...
##  $ Type.of.Travel        : int  1 2 2 2 2 1 1 2 2 2 ...
##  $ Class                 : int  1 2 2 2 2 3 3 2 2 3 ...
##  $ Flight.Distance_log   : num  6.13 5.46 7.04 6.33 5.37 ...
##  $ Inflight.wifi.service : int  3 3 2 2 3 3 2 4 1 3 ...
##  $ Seat.comfort          : int  5 1 5 2 5 1 2 5 3 3 ...
##  $ Inflight.entertainment: int  5 1 5 2 3 1 2 5 1 2 ...
##  $ On.board.service      : int  4 1 4 2 3 3 3 5 1 2 ...
##  $ Cleanliness           : int  5 1 5 2 3 1 2 4 2 2 ...
##  - attr(*, "na.action")= 'omit' Named int [1:310] 214 1125 1530 2005 2109 2486 2631 3622 4042 4491 ...
##   ..- attr(*, "names")= chr [1:310] "214" "1125" "1530" "2005" ...
library(corrplot)
cor_matrix = cor(num_vars, use = "complete.obs")
corrplot(cor_matrix, method = "number")

Inflight.service는 On.board.service와 가장 큰 상관관계가 있다는 것을 알 수 있다. On.board.service를 Inflight.service의 예측변수로 이용해서 Polynomial model을 만들기로 한다.

ggplot(airline_train_set, aes(x=factor(On.board.service), fill=factor(Inflight.service))) +
  geom_bar(position="fill") +
  labs(x="On.board.service", y="Proportion", title="Proportion")

On.board.service의 만족도가 높을수록 Inflight.service역시 높은 것을 확인할 수 있다.

polynomial model : Inflight.service ~ On.board.service

#10-fold cv
ctrl = trainControl(method = "cv", number = 10)
max_degree = 10
cv_errors = rep(NA, max_degree)

model = train(Inflight.service ~ poly(On.board.service, 1, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[1] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 2, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[2] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 3, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[3] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 4, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[4] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 5, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[5] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 6, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[6] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 7, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[7] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 8, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[8] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 9, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[9] = mean(model$results$RMSE^2)
model = train(Inflight.service ~ poly(On.board.service, 10, raw=TRUE), data = airline_train_set, method="lm", trControl = ctrl)
cv_errors[10] = mean(model$results$RMSE^2)
# 교차 검증 오차를 그래프로 시각화
plot(1:max_degree, cv_errors, type = "b", 
     xlab = "Degree of Polynomial", ylab = "Cross-Validation Error",
     main = "Cross-Validation Error vs. Polynomial Degree")

차수가 4부터 cv-error가 큰 폭으로 감소하는 것을 확인할 수 있다. 해당 polynomial model의 차수는 4를 선택하기로 한다.

Evaluation : polynomial model(order : 4)

poly_model = lm(Inflight.service ~ poly(On.board.service, 4) ,data = airline_train_set)
summary(poly_model)
## 
## Call:
## lm(formula = Inflight.service ~ poly(On.board.service, 4), data = airline_train_set)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -6.3230 -0.5553  0.1252  0.4899  2.4118 
## 
## Coefficients:
##                              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                  3.640761   0.003026 1203.24  < 2e-16 ***
## poly(On.board.service, 4)1 208.382318   0.973886  213.97  < 2e-16 ***
## poly(On.board.service, 4)2  13.213559   0.973886   13.57  < 2e-16 ***
## poly(On.board.service, 4)3  -7.927480   0.973886   -8.14 3.99e-16 ***
## poly(On.board.service, 4)4  35.495545   0.973886   36.45  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9739 on 103589 degrees of freedom
## Multiple R-squared:  0.3138, Adjusted R-squared:  0.3137 
## F-statistic: 1.184e+04 on 4 and 103589 DF,  p-value: < 2.2e-16

R-squared의 설명력이 낮은 이유는 예측하려는 종속변수가 카테고리 변수이기 때문임을 알 수 이다. test set으로 모델을 평가할 때, 모델의 출력을 그대로 사용하는 것이 아니라, 변환하는 과정이 필요하다.

#evaluation : test set
prediction_poly = predict(poly_model, newdata = airline_test_set)
predicted_poly_class = round(prediction_poly)
actual_labels = airline_test_set$Inflight.service
predicted_labels = predicted_poly_class

predicted_labels = factor((predicted_labels), levels = c(0,1,2,3,4,5))

print(levels(predicted_labels))
## [1] "0" "1" "2" "3" "4" "5"
confusionMatrix(predicted_labels, factor(actual_labels))
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    0    1    2    3    4    5
##          0    0    0    0    0    0    0
##          1    0    0    0    0    0    0
##          2    0    0    0    0    0    0
##          3    0 1197 2012 1584 1350  421
##          4    0  406  636 3227 6902 2333
##          5    0  172  190  194 1100 4167
## 
## Overall Statistics
##                                           
##                Accuracy : 0.4887          
##                  95% CI : (0.4826, 0.4948)
##     No Information Rate : 0.3612          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.2721          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 0 Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity                NA  0.00000   0.0000  0.31648   0.7380   0.6021
## Specificity                 1  1.00000   1.0000  0.76156   0.6008   0.9127
## Pos Pred Value             NA      NaN      NaN  0.24132   0.5111   0.7156
## Neg Pred Value             NA  0.93144   0.8904  0.82299   0.8022   0.8628
## Prevalence                  0  0.06856   0.1096  0.19331   0.3612   0.2673
## Detection Rate              0  0.00000   0.0000  0.06118   0.2666   0.1609
## Detection Prevalence        0  0.00000   0.0000  0.25352   0.5216   0.2249
## Balanced Accuracy          NA  0.50000   0.5000  0.53902   0.6694   0.7574
Sensitivity(민감도)
  • Class 0 : NAN
  • Class 1, 2 : 0.0% -> 모델이 해당 클래스를 전혀 예측하지 못하고 있다.
  • Class 3 : 31.65% -> 해당 클래스를 올바르게 예측한 비율이 약 32%로 낮았다.
  • Class 4 : 73.80% -> 해당 클래스를 올바르게 예측한 비율이 약 74%이다.
  • Class 5 : 60.21% -> 해당 클래스를 올바르게 예측한 비율이 약 60%이다.
Specificity(특이도)
  • Class 0, 1, 2 : 100%
  • Class 3 : 76.1%
  • Class 4 : 60.0%
  • Class 5 : 91.2%
result

class 4를 제외한 나머지 class에 대해서는 낮은 성능을 보였다.
Inflight.service를 예측하는 데, polynomial model은 적절않다는 결론을 내린다. 정규화 항을 추가해보기로 한다.

polynomial model : Inflight.service ~ On.board.service + regularization Term(Ridge)

airline_test_set_filtered = airline_test_set[airline_test_set$Inflight.service != 0, ]
airline_train_set_filtered = airline_train_set[airline_train_set$Inflight.service != 0, ]
library(glmnet)
## Loading required package: Matrix
## Loaded glmnet 4.1-8
x = model.matrix( ~ poly(On.board.service, 4), data = airline_train_set_filtered)[, -1]
y = as.factor(airline_train_set_filtered$Inflight.service)

ridge_model = glmnet(x, y, family = "multinomial" ,alpha = 0)
# cv를 통해서 lambda 선택
cv_fit = cv.glmnet(x, y, family="multinomial", alpha=0, type.measure="class")
best_lambda = cv_fit$lambda.min 
print(best_lambda)
## [1] 0.04341833

적절한 람다값은 0.03956082 이다. 해당 값으로 모델을 fitting 하기로 한다.

y = as.integer(as.factor(airline_train_set_filtered$Inflight.service)) - 1
best_ridge_model = glmnet(x, y, alpha = 0, lambda = best_lambda)
newx_test = model.matrix( ~ poly(On.board.service, 4), data = airline_test_set_filtered)[, -1] 
predicted = predict(best_ridge_model, newx=newx_test, s="lambda.min", type="response")
predicted_class = pmax(0, pmin(5, predicted))
final_predicted = round(predicted_class)
final_predicted
##     [1] 4 3 3 1 1 3 4 3 1 3 2 1 4 4 3 3 3 2 4 3 4 3 1 1 2 2 4 1 4 2 2 1 3 4 4 2
##    [37] 3 4 3 2 2 3 4 4 1 4 4 4 4 3 1 4 3 1 3 1 3 2 1 3 3 4 4 2 4 3 3 4 4 3 1 2
##    [73] 1 3 2 4 3 4 3 1 2 1 3 2 3 1 1 4 4 3 4 4 3 1 3 1 4 3 1 2 1 4 1 4 1 2 3 2
##   [109] 3 4 1 1 3 1 2 3 2 3 3 3 2 4 4 2 4 1 2 1 3 3 2 4 4 2 4 2 3 3 4 3 3 3 3 3
##   [145] 1 3 3 3 4 1 1 3 2 1 4 3 1 3 3 1 4 2 4 4 1 4 1 1 4 3 4 2 3 3 4 4 3 4 3 1
##   [181] 2 1 2 1 4 3 1 3 3 2 4 4 1 3 1 2 2 1 2 3 1 3 1 3 3 4 1 1 1 4 1 3 1 3 3 1
##   [217] 2 3 2 4 2 3 3 3 1 1 2 3 1 3 2 4 4 3 3 2 2 3 1 3 4 2 4 4 1 3 2 2 1 4 4 3
##   [253] 1 3 2 4 3 3 2 3 2 4 1 2 4 1 2 4 4 3 2 3 2 2 4 3 3 2 2 1 2 1 1 1 3 1 2 4
##   [289] 4 3 1 2 2 1 2 4 2 2 2 4 2 4 4 3 2 3 4 1 4 2 1 4 4 3 3 1 3 4 4 2 2 4 2 4
##   [325] 3 3 4 2 2 1 1 1 1 2 2 3 4 4 1 1 2 3 1 2 3 1 2 3 2 1 3 1 2 2 1 4 4 2 3 4
##   [361] 2 2 4 2 1 3 3 1 4 2 3 1 1 1 3 4 1 1 4 3 1 1 1 3 4 3 1 3 1 4 3 4 3 3 2 3
##   [397] 4 1 1 2 1 3 1 3 2 1 3 2 1 2 1 2 3 2 3 4 4 3 4 2 3 4 1 2 4 1 2 1 1 4 4 1
##   [433] 4 4 3 3 4 1 4 1 2 1 2 1 3 2 3 4 2 3 3 4 1 2 1 1 1 3 3 2 1 2 2 4 1 4 4 3
##   [469] 4 4 3 3 1 3 1 1 3 4 2 3 4 3 3 2 3 3 3 3 3 2 3 1 2 1 4 3 2 4 3 3 3 3 2 1
##   [505] 3 4 1 4 3 3 3 1 1 1 2 1 1 3 1 1 2 2 1 3 2 3 2 3 1 3 1 4 2 2 1 3 2 3 3 1
##   [541] 3 4 3 4 4 3 3 3 3 3 3 3 3 4 1 4 2 4 1 3 2 3 4 1 3 1 4 2 1 2 2 3 4 4 1 1
##   [577] 2 2 1 4 3 3 2 4 3 3 2 1 3 4 2 2 1 3 1 1 1 1 4 1 2 2 3 2 2 3 3 1 3 2 2 1
##   [613] 1 3 3 4 3 2 1 2 2 4 2 4 2 4 2 3 2 2 4 3 1 2 2 4 2 2 2 3 3 1 1 3 3 2 1 3
##   [649] 1 2 1 1 1 1 4 3 2 3 4 3 2 2 1 1 3 2 3 4 1 1 3 3 1 4 3 2 3 3 2 3 4 2 3 4
##   [685] 1 3 2 3 3 1 3 3 4 4 3 1 1 2 4 4 1 2 3 3 4 3 1 3 4 3 1 3 1 3 3 1 1 4 3 3
##   [721] 3 4 4 3 2 3 3 3 4 4 2 3 4 1 1 1 1 2 3 3 3 3 4 3 3 3 1 3 3 3 4 2 4 3 1 2
##   [757] 3 1 3 1 3 1 2 3 1 1 3 4 1 2 2 2 4 1 3 2 2 1 4 1 2 3 4 1 4 1 2 3 1 4 4 1
##   [793] 1 1 3 2 3 3 1 2 4 1 1 4 2 1 4 3 1 3 1 1 3 1 3 1 2 2 2 2 1 2 3 3 4 1 4 4
##   [829] 3 1 3 4 2 3 3 4 2 1 4 1 2 3 3 1 1 4 3 1 3 4 3 4 4 2 3 4 1 3 3 1 3 1 3 2
##   [865] 3 1 3 2 2 1 2 3 4 4 4 2 3 2 4 3 2 1 3 3 2 1 4 3 1 3 1 1 1 3 3 1 4 3 4 3
##   [901] 1 3 3 2 3 1 3 1 2 1 3 2 1 4 4 4 4 1 4 1 4 2 2 3 3 2 3 3 2 3 1 2 4 4 4 4
##   [937] 2 2 4 3 4 3 2 4 4 2 2 3 4 1 1 4 2 2 2 3 2 3 1 4 3 2 2 2 2 1 4 1 4 3 1 1
##   [973] 3 1 1 1 4 1 3 1 4 2 1 2 2 3 2 1 3 3 2 4 3 2 2 3 3 2 2 2 3 4 4 1 3 4 1 1
##  [1009] 3 4 2 3 2 3 3 3 2 1 1 4 2 1 3 4 1 3 4 3 2 1 4 3 1 2 2 4 3 2 1 2 1 3 1 2
##  [1045] 3 3 4 3 4 1 1 3 3 3 1 4 1 1 3 3 4 4 4 4 2 2 3 2 3 1 1 3 3 1 3 1 1 3 1 4
##  [1081] 1 2 2 2 4 4 1 2 2 3 2 4 1 2 3 1 1 1 2 4 4 2 3 2 1 1 2 3 2 3 3 3 4 3 3 1
##  [1117] 1 2 1 4 1 4 4 4 3 3 1 2 4 3 3 3 1 2 3 3 1 1 3 1 3 4 3 4 1 3 3 2 4 4 4 1
##  [1153] 3 4 2 2 4 2 4 4 3 4 3 3 3 1 3 4 3 1 3 3 2 4 1 1 1 2 4 3 2 1 1 3 3 1 3 4
##  [1189] 1 2 3 3 4 2 3 3 1 4 3 3 3 4 3 1 4 2 2 1 4 1 1 3 3 2 1 2 4 3 1 1 3 1 2 2
##  [1225] 1 2 3 4 4 3 2 3 1 1 1 3 1 2 4 4 4 3 3 1 4 3 4 3 3 4 4 2 3 1 4 3 1 4 2 2
##  [1261] 2 3 1 3 1 4 4 3 4 2 3 1 1 1 1 1 3 3 4 3 2 1 3 4 2 1 2 3 1 3 4 2 2 3 2 1
##  [1297] 2 3 2 2 3 2 1 4 3 1 1 4 1 2 3 3 1 4 3 3 3 3 1 3 1 3 4 4 2 2 3 2 2 4 2 1
##  [1333] 3 3 2 3 1 1 4 4 3 4 3 2 2 3 1 4 4 4 4 2 1 1 3 3 3 4 3 2 4 3 3 3 1 1 2 4
##  [1369] 4 2 2 1 4 3 4 1 3 3 2 1 4 4 4 3 2 4 3 4 2 1 4 3 1 2 4 3 1 1 3 1 1 4 2 1
##  [1405] 1 4 2 3 4 1 1 3 2 4 2 1 2 3 4 4 1 3 1 1 3 3 3 1 4 4 2 3 3 4 2 1 1 3 3 3
##  [1441] 4 4 2 3 3 4 4 4 4 3 1 2 2 3 4 3 3 1 1 3 3 3 2 4 1 1 4 2 1 3 1 4 1 3 4 4
##  [1477] 2 1 3 4 4 3 3 4 4 2 3 3 3 3 3 4 1 2 1 2 1 1 3 2 3 4 3 1 4 2 1 3 1 4 1 2
##  [1513] 1 4 1 2 3 2 2 1 3 3 2 3 2 3 4 4 1 3 2 4 2 4 3 2 1 1 3 3 1 3 2 1 2 3 2 3
##  [1549] 3 2 4 1 3 3 4 3 4 3 2 4 3 3 3 3 3 2 4 1 4 2 4 2 1 2 1 4 1 2 3 4 2 4 1 3
##  [1585] 3 4 1 2 1 3 1 4 1 3 4 2 2 3 1 1 2 4 2 1 2 3 1 3 3 2 1 4 3 3 1 2 4 1 3 3
##  [1621] 1 2 2 1 2 1 1 1 3 1 4 1 3 3 3 3 2 2 1 4 4 2 3 3 4 3 1 1 4 3 1 3 4 2 4 3
##  [1657] 3 4 4 1 3 2 1 1 3 4 1 1 1 2 3 2 1 3 1 3 1 3 3 1 4 2 1 2 4 4 3 3 2 3 3 1
##  [1693] 2 1 2 1 1 3 2 1 1 1 4 4 4 4 1 3 1 2 2 1 2 2 2 3 4 2 2 1 1 4 3 1 3 4 4 2
##  [1729] 1 1 2 3 3 4 1 1 4 3 1 4 4 1 1 3 1 2 3 2 3 2 1 1 1 2 1 1 4 1 3 3 2 4 1 1
##  [1765] 3 3 3 1 4 4 2 1 2 2 2 3 2 1 3 3 3 2 4 2 3 2 2 3 3 4 3 4 3 3 2 4 2 1 1 3
##  [1801] 4 1 3 4 3 1 3 1 2 3 4 3 1 3 4 1 4 2 4 2 2 3 3 2 1 3 4 2 1 1 3 3 4 1 3 1
##  [1837] 4 1 2 3 4 4 3 4 1 1 1 3 3 4 1 3 4 3 3 3 1 4 3 4 3 2 4 4 3 4 2 2 4 3 2 1
##  [1873] 3 2 1 3 1 4 4 2 1 2 3 4 2 4 1 4 2 3 2 3 3 1 1 1 1 4 3 2 3 3 4 4 2 2 1 1
##  [1909] 3 1 4 3 1 3 4 3 2 1 4 4 4 3 1 3 1 3 2 3 4 2 1 1 4 3 1 2 1 4 3 3 2 4 3 2
##  [1945] 3 4 3 1 1 1 4 1 4 3 4 4 3 1 3 1 3 4 4 3 2 1 1 2 1 4 1 1 3 4 1 2 3 2 3 4
##  [1981] 1 3 3 3 4 1 2 1 4 2 2 2 1 4 3 2 1 3 1 1 2 3 1 1 4 3 4 2 4 4 3 3 4 3 3 3
##  [2017] 2 2 3 3 1 1 1 3 4 1 4 4 4 4 1 2 3 3 3 4 1 1 2 3 2 4 4 3 3 3 3 4 3 3 1 3
##  [2053] 4 4 4 3 2 2 3 2 4 1 3 4 3 2 4 1 2 3 1 1 3 1 1 4 2 3 2 1 4 2 3 2 1 1 2 3
##  [2089] 2 1 3 4 3 4 3 4 1 2 3 1 2 4 4 2 1 4 2 3 2 4 2 2 3 1 3 3 2 3 4 3 3 1 2 1
##  [2125] 3 1 2 1 4 2 4 3 4 4 1 4 3 3 2 1 3 2 1 2 4 4 1 1 2 4 2 1 3 2 3 2 4 3 4 4
##  [2161] 3 1 4 4 2 3 2 1 2 4 2 1 1 4 1 3 1 2 2 3 3 2 1 4 2 1 2 4 2 1 2 2 4 3 2 3
##  [2197] 1 3 2 1 2 1 1 4 3 2 1 3 1 2 2 4 4 4 3 2 1 4 3 2 3 2 3 1 2 3 3 3 4 2 4 4
##  [2233] 4 2 4 1 3 2 4 3 2 1 1 1 2 2 1 2 2 2 3 3 1 3 2 4 4 2 2 3 4 4 4 4 3 3 3 2
##  [2269] 2 3 2 2 1 1 3 4 4 3 3 1 1 2 4 4 1 1 3 3 2 3 1 3 4 2 3 1 2 1 1 2 2 1 2 4
##  [2305] 2 4 3 1 2 4 4 2 4 3 3 1 4 4 2 3 3 2 4 3 4 1 3 1 2 2 2 2 2 3 1 3 4 3 3 3
##  [2341] 4 2 2 3 3 2 3 1 1 1 3 1 4 4 1 3 4 2 2 4 2 3 1 2 1 1 3 1 1 3 3 4 3 3 1 3
##  [2377] 3 3 4 4 1 3 3 1 1 1 4 3 3 3 1 1 3 2 1 4 4 4 4 4 4 4 3 1 4 4 2 2 1 2 4 3
##  [2413] 1 2 1 2 4 1 3 4 3 4 1 3 1 4 4 2 3 2 3 4 3 1 4 4 1 2 4 3 4 2 4 1 4 2 3 4
##  [2449] 1 3 2 3 4 2 1 2 2 4 4 3 2 2 1 2 1 3 3 3 1 4 1 4 3 2 3 3 1 4 3 3 1 2 2 4
##  [2485] 1 2 1 1 2 4 1 4 3 1 3 1 1 3 1 4 4 4 3 2 1 4 3 4 3 4 3 4 1 3 3 4 4 3 1 1
##  [2521] 2 3 2 1 3 1 3 3 3 3 1 4 4 3 3 4 3 1 1 2 3 4 3 1 4 2 3 3 3 3 4 3 2 4 1 3
##  [2557] 4 1 3 1 3 2 2 1 4 4 3 1 3 2 2 1 3 2 1 3 1 2 3 3 3 1 3 1 1 1 2 3 3 4 2 2
##  [2593] 1 1 3 1 4 3 2 3 1 2 4 1 2 3 1 3 1 2 2 2 1 2 2 3 3 3 3 3 2 2 1 1 1 2 2 3
##  [2629] 3 1 4 3 4 3 3 1 1 4 4 1 2 4 2 3 1 3 1 3 1 4 3 1 1 1 4 1 1 1 1 2 3 1 4 4
##  [2665] 3 3 2 1 1 3 4 2 1 3 2 4 1 2 4 2 1 2 2 4 2 3 2 2 4 1 1 4 3 1 3 2 1 4 3 3
##  [2701] 1 3 3 2 2 3 4 3 1 1 2 2 3 3 3 4 2 3 1 3 4 1 3 2 4 1 1 3 1 2 2 4 4 1 4 2
##  [2737] 1 3 3 4 3 1 3 2 4 1 3 3 3 3 3 1 2 2 2 1 3 3 1 1 3 1 2 4 3 2 2 3 3 4 2 4
##  [2773] 3 2 3 2 2 1 2 3 2 1 3 4 2 2 2 2 2 2 1 3 1 3 4 2 2 2 4 1 2 4 3 1 2 1 3 3
##  [2809] 1 2 4 1 2 3 3 1 3 3 1 2 1 1 3 4 4 2 3 4 2 2 4 2 2 4 3 3 1 4 4 4 3 1 4 2
##  [2845] 1 1 2 3 3 2 4 1 3 1 2 2 3 3 4 3 3 2 1 4 4 4 2 3 3 1 4 3 1 4 1 2 2 1 3 4
##  [2881] 2 4 3 4 1 4 2 2 3 1 1 3 3 4 3 2 1 4 3 1 4 1 1 3 4 3 2 3 3 4 3 1 3 1 3 3
##  [2917] 4 4 1 3 1 3 4 4 4 1 2 4 2 4 2 3 1 2 1 1 2 1 1 1 2 1 2 4 1 3 3 3 1 4 1 2
##  [2953] 1 2 2 2 4 4 4 4 2 4 2 2 3 3 3 2 2 2 4 3 4 3 3 1 1 4 2 1 3 4 4 3 2 4 3 2
##  [2989] 1 3 1 3 3 3 4 4 4 2 2 2 1 4 2 3 4 3 1 1 1 4 1 2 1 1 1 3 1 3 1 1 1 3 2 2
##  [3025] 4 3 3 3 1 3 4 4 4 1 3 2 3 3 4 2 1 2 3 3 4 1 1 3 3 3 4 2 1 4 2 3 3 1 2 2
##  [3061] 1 3 2 2 3 2 2 3 2 2 3 3 3 3 3 3 3 3 4 1 4 4 3 1 3 3 3 3 2 3 4 4 1 3 4 1
##  [3097] 2 3 1 2 4 3 3 1 3 3 3 1 3 3 1 1 4 2 3 1 4 2 2 2 3 2 2 1 2 3 4 4 4 1 2 3
##  [3133] 1 3 2 1 1 2 3 1 3 4 1 2 1 1 1 2 3 4 3 1 1 4 2 4 3 3 3 3 3 4 3 4 1 1 3 2
##  [3169] 2 1 1 3 4 2 1 1 3 3 3 4 3 3 3 2 1 1 2 2 1 2 3 2 3 4 1 3 1 2 1 3 1 3 3 2
##  [3205] 3 3 1 1 1 3 3 2 1 3 4 2 1 3 2 4 4 4 4 1 3 3 4 3 4 1 1 1 1 3 3 2 2 4 1 4
##  [3241] 3 3 3 2 4 4 3 1 2 3 2 2 2 1 4 3 2 2 4 1 1 1 2 3 3 2 3 2 4 3 1 2 1 1 1 2
##  [3277] 2 1 1 1 2 1 3 3 1 4 4 3 3 1 3 1 1 1 3 4 3 3 1 3 1 1 4 2 1 2 3 4 4 1 2 1
##  [3313] 1 2 1 2 3 1 3 2 4 2 3 4 3 3 4 2 4 3 4 3 2 3 1 2 4 1 1 1 2 1 2 4 3 1 4 3
##  [3349] 3 4 1 2 2 2 2 3 3 2 3 2 4 3 1 2 4 1 4 1 4 1 1 3 3 4 3 3 1 3 1 4 1 1 1 3
##  [3385] 1 1 4 1 3 1 3 4 1 4 1 2 4 4 3 2 2 3 3 1 2 3 4 3 4 1 2 3 4 3 1 1 4 1 1 3
##  [3421] 4 3 1 1 1 1 4 1 3 3 3 3 1 2 4 1 1 3 3 3 3 2 4 3 3 3 2 3 3 2 3 4 1 1 1 4
##  [3457] 1 2 4 2 3 3 3 4 1 1 1 3 4 3 4 2 3 4 3 2 3 3 1 1 2 2 1 3 1 2 3 3 4 1 3 3
##  [3493] 3 4 2 1 4 1 3 4 1 4 1 3 1 4 3 2 2 3 1 3 1 3 3 1 3 4 4 3 1 3 3 3 1 4 3 3
##  [3529] 1 3 1 3 3 4 1 4 3 2 4 1 2 3 1 2 3 2 2 4 3 2 1 4 2 4 1 4 2 3 1 1 3 1 2 2
##  [3565] 3 4 4 2 4 4 3 2 4 4 1 2 3 3 4 4 4 3 1 4 3 4 2 1 2 3 1 3 3 1 1 3 2 1 3 3
##  [3601] 3 3 2 4 1 3 2 4 3 2 1 4 2 2 2 1 1 4 1 2 2 3 4 4 4 2 4 2 1 4 2 3 2 3 4 1
##  [3637] 1 1 2 3 1 1 4 1 3 1 1 1 3 1 2 2 2 3 3 1 1 3 3 4 2 1 1 3 3 3 4 2 1 3 1 4
##  [3673] 3 2 1 1 3 1 4 2 3 3 1 3 3 3 4 3 2 3 4 2 4 1 1 3 3 3 1 2 1 3 1 1 3 2 3 1
##  [3709] 1 3 4 1 3 2 3 2 1 4 1 2 2 2 3 4 1 3 3 4 4 1 2 3 1 2 1 4 3 1 1 3 3 1 1 2
##  [3745] 2 3 3 3 1 3 4 1 4 2 3 1 2 3 2 3 3 1 3 4 1 2 2 3 1 2 4 1 4 2 3 2 3 1 1 1
##  [3781] 4 2 2 1 1 4 2 1 2 2 2 4 1 2 2 2 1 3 3 2 1 2 4 3 1 3 3 3 2 3 4 3 3 1 2 4
##  [3817] 2 3 2 1 2 1 4 3 4 2 2 3 4 1 4 3 4 1 3 1 1 1 3 4 1 2 4 3 2 3 4 2 3 1 2 2
##  [3853] 1 3 1 4 4 3 1 4 1 4 1 3 2 3 4 1 4 1 4 1 1 1 3 4 3 1 3 1 3 1 1 3 3 3 3 2
##  [3889] 2 4 4 3 4 3 3 2 2 3 2 3 4 2 2 3 4 2 3 3 3 1 4 4 4 2 4 2 2 4 4 4 4 4 3 3
##  [3925] 3 3 4 1 1 1 3 3 4 2 4 2 1 1 2 2 3 3 3 1 4 3 3 1 3 4 3 1 1 1 1 1 3 2 3 3
##  [3961] 2 3 3 4 4 3 2 3 1 1 3 4 3 2 2 3 2 2 3 4 4 2 4 3 1 4 1 1 3 3 1 4 3 4 3 3
##  [3997] 3 4 3 3 4 2 3 1 1 4 2 3 1 3 2 3 4 1 3 1 1 1 1 3 3 3 4 2 1 1 1 3 2 2 2 2
##  [4033] 1 3 4 1 3 3 3 4 4 3 3 2 3 3 1 2 1 3 4 2 4 4 1 1 3 4 1 3 3 1 4 1 3 2 3 3
##  [4069] 2 1 1 4 1 1 3 3 1 2 3 2 4 3 3 3 1 2 2 1 1 4 1 4 1 1 4 4 4 4 2 4 3 1 3 3
##  [4105] 3 1 2 3 4 4 4 1 3 4 2 3 3 1 1 2 3 3 3 3 1 1 3 1 3 1 3 2 4 2 4 4 3 2 4 3
##  [4141] 1 2 4 2 4 3 3 1 1 3 1 2 1 3 1 1 4 3 1 4 1 3 1 1 2 1 2 4 4 3 1 4 3 2 2 3
##  [4177] 1 4 3 2 1 4 2 2 1 4 4 4 1 4 3 4 1 2 3 3 2 3 4 4 3 2 4 2 4 1 3 1 3 3 4 4
##  [4213] 2 1 2 3 1 3 3 2 1 2 1 4 4 1 2 2 4 3 3 2 2 4 3 3 2 2 1 1 2 4 1 3 4 4 2 2
##  [4249] 2 1 2 2 2 1 2 1 3 4 4 2 4 3 3 4 1 2 4 1 4 2 2 2 1 1 3 2 4 4 4 3 1 2 1 1
##  [4285] 2 3 3 2 2 1 1 4 3 2 4 4 3 2 3 1 2 2 1 3 2 4 4 1 4 4 3 2 2 4 1 3 2 3 4 1
##  [4321] 4 3 4 2 4 1 3 3 4 3 3 1 4 3 3 3 3 3 1 4 3 2 2 4 3 3 4 1 2 4 3 4 4 3 1 3
##  [4357] 2 2 3 1 3 4 4 1 2 4 2 4 3 1 4 3 3 3 2 1 3 3 1 3 1 4 2 2 4 2 1 4 3 3 1 1
##  [4393] 2 3 3 2 1 1 4 4 1 4 2 1 1 4 3 3 4 4 1 4 2 1 1 4 2 2 3 1 4 4 4 2 4 3 4 4
##  [4429] 1 1 1 2 3 3 2 1 4 4 3 4 1 1 1 1 1 1 3 3 3 4 4 3 4 4 4 3 3 2 3 1 1 3 3 3
##  [4465] 1 3 3 3 3 3 4 1 3 2 4 3 3 3 2 2 2 2 2 2 3 3 2 1 1 1 4 4 3 3 3 4 1 3 3 4
##  [4501] 1 1 3 3 2 1 1 2 3 3 1 4 4 1 3 1 2 4 1 3 3 4 1 2 1 1 2 1 4 3 2 2 3 2 2 4
##  [4537] 2 3 2 4 3 2 1 2 1 1 2 1 3 3 3 4 4 4 4 3 4 2 3 4 2 3 1 4 3 4 2 3 4 3 4 4
##  [4573] 3 3 4 3 2 1 2 3 1 1 1 4 1 4 3 1 4 3 3 4 3 3 1 2 3 1 4 2 3 1 4 1 2 3 2 1
##  [4609] 3 3 2 3 1 3 1 3 2 3 1 3 2 1 2 3 2 2 1 2 1 2 3 2 4 2 4 4 4 2 4 4 3 3 3 2
##  [4645] 2 1 2 1 1 2 4 3 2 3 1 2 4 3 3 3 1 4 3 2 2 3 3 1 1 3 3 1 1 1 4 3 4 1 4 4
##  [4681] 1 4 1 4 4 3 3 4 2 2 4 3 1 3 1 3 3 4 4 1 2 2 2 4 1 1 2 3 1 1 3 1 1 1 4 4
##  [4717] 2 4 2 2 4 2 1 4 3 3 1 1 2 1 1 4 3 3 2 1 3 3 3 2 3 3 3 4 4 3 2 3 4 1 3 2
##  [4753] 2 3 1 3 1 2 2 2 3 2 2 2 3 4 2 2 3 3 4 3 2 3 2 1 1 1 3 1 1 4 2 4 1 4 4 4
##  [4789] 3 2 4 1 2 2 2 2 2 2 1 3 2 4 1 2 2 3 3 4 2 3 1 2 4 4 3 4 4 1 3 1 1 4 2 1
##  [4825] 1 3 2 3 4 4 2 1 3 1 4 1 2 4 2 2 4 1 1 4 1 3 4 4 3 1 4 3 3 3 1 2 1 3 1 2
##  [4861] 1 3 4 4 1 2 4 3 1 3 4 2 3 3 3 4 2 4 2 1 3 4 1 2 4 4 3 3 4 3 2 3 4 2 2 4
##  [4897] 4 3 4 2 4 1 1 3 4 1 3 3 3 3 3 1 1 2 1 3 1 2 2 2 1 4 3 4 3 3 4 3 1 2 1 4
##  [4933] 2 2 2 4 2 2 4 2 2 3 3 3 3 3 4 3 2 2 1 3 3 4 3 2 1 4 3 1 1 3 1 1 1 3 1 2
##  [4969] 3 1 2 3 1 2 2 2 2 4 4 4 4 1 4 4 4 3 4 4 2 3 3 4 3 1 2 2 3 3 4 2 3 3 4 4
##  [5005] 1 4 4 2 2 1 2 2 4 3 1 3 3 4 4 4 2 1 2 1 3 4 3 4 3 2 4 4 2 4 4 3 1 1 3 1
##  [5041] 3 4 1 2 3 3 3 3 3 1 1 2 1 1 1 4 3 1 3 2 3 3 2 3 2 1 3 3 4 3 2 2 3 1 3 3
##  [5077] 3 4 3 3 1 3 1 2 3 2 2 3 3 1 3 4 2 2 3 2 1 2 4 2 2 4 3 1 2 4 4 2 1 2 3 2
##  [5113] 3 1 2 1 1 3 4 3 4 2 3 3 3 4 1 4 1 3 3 4 3 1 4 3 4 4 4 2 1 2 2 2 3 3 3 2
##  [5149] 3 4 3 3 3 3 1 1 2 1 4 2 3 1 2 4 1 1 2 2 1 3 2 3 1 4 3 4 3 3 3 4 4 4 3 3
##  [5185] 3 1 3 4 1 3 2 3 1 4 1 2 2 1 3 2 3 4 2 1 2 4 4 1 1 4 2 4 1 4 3 2 3 3 4 1
##  [5221] 1 4 2 3 3 3 2 2 3 3 3 1 1 3 4 3 1 2 2 2 2 2 1 3 4 4 2 3 3 3 1 4 3 2 4 2
##  [5257] 3 1 4 2 3 4 2 1 1 3 1 1 2 2 1 2 1 2 3 2 2 3 4 2 2 4 3 1 1 3 3 3 2 1 3 4
##  [5293] 3 2 3 3 4 1 3 4 4 3 1 4 4 1 2 2 3 3 4 4 2 1 3 2 3 4 4 2 2 2 3 3 1 2 4 2
##  [5329] 4 1 4 1 2 2 3 3 3 1 3 4 1 1 4 2 2 1 3 3 1 4 3 2 3 1 1 3 4 3 3 3 4 1 3 4
##  [5365] 3 3 1 1 1 4 4 4 1 4 4 2 4 1 2 3 4 1 3 3 2 1 1 4 2 3 1 1 4 3 3 3 2 2 4 2
##  [5401] 2 2 2 3 4 3 3 1 3 1 4 1 3 3 3 4 2 2 1 3 1 1 2 4 4 4 1 2 1 1 2 2 1 4 2 2
##  [5437] 4 1 3 4 3 4 1 4 4 2 4 4 2 1 4 4 1 2 4 2 2 2 4 3 4 4 1 4 4 3 4 1 2 3 2 4
##  [5473] 2 1 3 4 2 1 2 4 1 1 2 4 2 3 1 1 1 3 1 1 4 4 2 4 1 1 4 2 1 4 4 4 4 3 3 4
##  [5509] 4 2 1 1 3 3 4 1 3 1 4 4 1 3 3 2 1 3 3 1 3 4 4 2 2 3 2 1 3 2 3 2 3 4 3 2
##  [5545] 1 4 3 1 2 4 3 1 3 2 1 3 4 3 3 3 3 1 4 1 3 4 2 1 1 4 2 4 2 3 3 3 3 2 2 3
##  [5581] 4 4 4 2 3 4 2 1 3 2 2 2 1 3 3 2 2 1 2 3 2 3 4 4 1 3 3 1 1 4 1 2 4 4 1 2
##  [5617] 1 3 4 2 3 3 2 1 2 2 2 1 3 3 3 4 4 2 3 2 4 4 4 4 4 1 2 3 3 1 4 4 4 3 1 3
##  [5653] 4 4 2 3 4 4 4 1 1 2 1 4 4 3 1 1 2 2 1 1 1 4 3 1 4 4 3 2 2 3 4 1 3 3 1 3
##  [5689] 3 1 3 1 2 1 1 3 4 3 4 2 3 3 4 4 3 4 3 2 4 4 1 4 3 1 4 1 1 4 1 2 1 1 4 4
##  [5725] 3 1 4 1 3 2 3 1 4 3 3 4 2 1 2 3 2 2 2 2 1 2 4 1 1 3 4 3 3 2 3 3 4 3 3 3
##  [5761] 3 1 1 2 3 3 2 3 3 3 3 2 4 2 4 4 2 3 1 3 4 3 1 2 4 1 4 3 1 1 2 3 2 2 3 4
##  [5797] 1 1 1 3 3 1 3 4 4 4 4 3 2 4 1 1 4 4 4 3 4 3 3 1 3 1 4 4 3 3 3 1 1 3 1 2
##  [5833] 3 3 1 2 3 3 2 4 3 2 4 2 4 2 4 3 3 2 3 1 3 3 3 2 2 4 2 3 4 3 3 2 1 1 4 3
##  [5869] 4 2 4 4 1 3 2 3 1 3 3 1 4 3 1 3 2 2 1 2 4 4 3 4 4 3 1 1 3 3 4 3 1 3 1 3
##  [5905] 4 3 4 3 4 1 1 3 1 3 2 4 3 1 3 2 1 4 4 2 4 3 3 2 4 4 3 1 1 3 2 3 2 3 2 1
##  [5941] 4 1 1 3 3 2 2 3 3 1 1 1 2 3 4 1 4 3 3 2 2 1 1 4 4 4 4 2 1 1 2 3 2 4 1 1
##  [5977] 3 3 3 1 4 2 2 2 2 3 1 2 3 1 2 4 4 3 3 3 2 1 2 3 1 1 2 2 2 4 4 2 3 2 2 4
##  [6013] 1 3 4 2 3 1 4 3 4 4 3 2 1 1 2 3 4 1 1 3 2 4 2 1 1 1 3 2 1 4 1 3 4 4 3 4
##  [6049] 1 1 4 1 1 4 1 1 3 4 3 1 3 2 2 1 3 2 3 2 1 3 2 1 4 1 3 3 1 1 4 3 4 4 2 3
##  [6085] 3 3 1 1 1 1 4 4 4 3 1 3 2 4 3 1 3 4 1 3 2 3 1 3 4 3 1 3 2 4 2 1 3 1 3 2
##  [6121] 3 1 1 2 4 3 1 3 1 4 1 3 3 3 3 4 2 3 3 1 1 4 4 4 2 3 2 2 1 1 3 2 3 1 1 1
##  [6157] 1 4 1 2 1 3 2 1 2 4 4 3 2 3 1 4 4 2 4 4 3 3 1 1 2 4 4 3 2 4 4 3 1 2 2 4
##  [6193] 3 3 3 2 2 3 4 4 3 1 1 2 4 3 1 1 3 3 3 4 3 1 3 4 4 3 2 2 2 4 2 4 1 2 3 3
##  [6229] 1 3 1 2 4 2 2 1 1 3 2 3 1 1 4 3 2 4 4 2 1 2 2 3 4 2 3 2 2 3 2 1 4 1 3 2
##  [6265] 3 1 1 3 2 4 2 2 3 3 1 3 3 2 4 4 1 2 1 1 4 2 1 2 4 1 1 4 4 1 1 1 4 3 2 1
##  [6301] 3 3 2 3 3 1 2 3 2 3 3 3 3 1 2 2 1 2 1 1 3 1 3 4 2 1 4 4 1 4 2 4 4 2 4 1
##  [6337] 3 4 1 4 1 2 4 3 1 3 3 3 1 3 4 3 2 3 2 2 3 1 2 3 1 4 1 3 4 2 3 1 3 4 1 4
##  [6373] 2 4 4 3 3 4 1 3 2 2 2 4 3 4 1 3 3 1 1 1 2 3 4 3 2 4 2 1 1 4 3 2 1 4 1 3
##  [6409] 3 4 1 1 2 4 3 2 2 3 3 1 2 2 1 2 3 4 4 3 4 2 1 3 1 4 4 1 3 2 3 1 3 3 2 3
##  [6445] 1 1 1 4 3 3 1 3 4 3 3 1 1 4 4 3 1 4 2 3 2 1 3 1 3 3 4 3 3 3 3 3 4 2 4 3
##  [6481] 2 4 1 3 1 2 1 4 2 3 4 4 3 2 2 1 3 3 4 1 3 1 4 3 2 4 4 3 4 4 3 3 2 1 3 2
##  [6517] 3 3 1 2 3 3 4 3 1 4 4 4 2 4 3 4 1 2 2 1 3 2 2 2 3 3 4 3 4 1 2 2 3 4 4 4
##  [6553] 3 3 2 3 1 2 1 3 4 2 3 2 4 2 3 1 3 3 1 4 4 1 1 3 2 4 1 3 1 1 4 3 3 4 2 4
##  [6589] 4 2 2 4 4 4 4 3 2 1 3 1 1 2 3 3 1 2 2 4 2 1 1 1 4 3 1 3 1 2 2 2 3 4 2 1
##  [6625] 4 3 3 1 3 1 3 1 1 4 2 4 4 2 4 3 4 2 2 4 4 3 1 2 4 2 2 3 3 3 2 3 2 3 1 3
##  [6661] 1 1 4 2 1 3 3 1 1 4 1 4 3 2 1 2 1 2 1 2 2 4 1 4 4 3 2 1 4 2 4 1 1 3 1 2
##  [6697] 3 3 1 4 4 1 4 1 4 3 3 1 3 4 3 3 1 4 3 4 3 4 4 4 1 1 1 3 3 4 1 3 4 2 1 3
##  [6733] 4 2 1 4 1 3 1 3 4 4 4 1 2 1 1 2 4 4 3 4 1 4 3 1 2 4 3 2 2 4 3 3 1 4 2 4
##  [6769] 3 4 4 2 4 2 2 2 4 2 4 2 3 4 3 1 4 2 2 3 1 3 4 3 1 3 1 1 2 1 2 4 3 1 1 4
##  [6805] 2 2 2 4 3 3 3 3 3 2 3 2 3 4 3 4 2 1 1 4 2 3 1 2 1 1 2 1 1 3 4 4 3 2 4 3
##  [6841] 3 2 4 4 2 3 3 3 2 2 1 3 1 2 4 3 2 3 4 4 2 2 3 3 2 1 3 3 2 3 3 2 1 1 3 1
##  [6877] 1 3 4 4 3 1 4 2 3 3 2 2 1 1 1 3 1 2 4 2 3 2 4 2 1 1 2 4 3 1 3 2 3 2 4 1
##  [6913] 2 1 2 4 4 3 1 3 2 3 3 3 4 1 2 3 1 4 4 2 4 1 2 1 3 3 3 3 2 2 2 3 4 4 3 3
##  [6949] 2 1 1 4 2 2 2 2 1 4 2 2 4 1 3 4 4 2 3 3 3 2 2 4 4 2 2 2 1 1 3 2 2 1 1 2
##  [6985] 2 1 2 2 4 4 4 1 3 1 1 4 4 3 3 4 1 3 4 3 4 4 1 3 4 3 2 3 1 1 3 2 3 3 2 2
##  [7021] 1 3 3 1 3 1 2 3 1 1 3 2 2 1 1 4 1 4 3 3 3 2 3 3 3 3 1 2 4 3 4 1 4 3 1 2
##  [7057] 3 2 2 3 3 3 3 2 2 3 4 2 2 1 1 2 1 4 2 1 1 3 3 3 3 4 3 3 3 3 1 1 1 1 1 1
##  [7093] 4 4 2 3 3 2 1 4 1 4 3 4 1 3 3 3 3 1 3 3 3 3 2 1 1 3 3 2 3 1 3 1 4 3 2 1
##  [7129] 1 1 4 4 3 3 1 4 2 4 3 2 3 4 4 1 4 3 4 2 4 3 2 4 3 1 3 3 3 1 2 4 3 2 1 1
##  [7165] 4 3 2 3 1 1 2 1 2 1 2 1 1 2 4 2 1 4 1 2 2 1 1 4 2 1 4 4 1 2 3 3 4 1 4 3
##  [7201] 3 1 4 3 1 4 2 4 2 4 1 4 2 3 3 4 1 1 4 2 4 3 2 1 3 1 2 1 4 2 3 4 2 3 1 3
##  [7237] 1 1 4 4 1 1 3 2 3 1 2 4 4 4 2 3 1 1 2 4 3 3 3 2 3 4 3 1 3 1 2 3 1 2 3 1
##  [7273] 3 4 4 2 2 1 3 2 1 4 2 3 1 3 4 3 1 4 1 2 3 3 3 3 4 1 2 1 3 3 2 4 3 1 3 3
##  [7309] 4 4 4 4 1 3 4 1 1 2 3 2 3 2 4 3 2 1 1 3 3 1 3 3 3 3 1 2 2 2 3 2 2 1 2 2
##  [7345] 2 1 4 1 4 2 3 3 4 2 1 1 4 4 4 1 1 3 3 2 3 2 3 3 3 3 3 2 3 3 4 4 4 4 3 2
##  [7381] 1 3 3 1 2 1 1 3 4 1 2 3 2 1 2 3 3 4 4 4 4 1 1 1 3 2 3 3 1 4 2 3 1 4 2 2
##  [7417] 2 1 4 1 3 1 3 1 1 2 2 2 1 2 3 2 1 4 4 3 2 3 1 3 3 2 2 1 3 2 3 3 1 2 3 3
##  [7453] 1 3 1 2 3 3 3 3 4 4 3 4 2 1 4 3 4 2 2 1 3 3 1 4 2 1 4 1 1 1 3 1 1 4 3 4
##  [7489] 3 3 2 2 3 3 3 4 4 4 2 4 3 1 1 3 1 2 2 1 4 1 4 2 3 3 3 2 4 4 2 3 2 2 4 1
##  [7525] 1 1 1 4 3 4 1 3 3 4 2 4 2 1 2 1 3 3 1 4 1 3 1 4 1 4 3 1 2 3 3 3 4 3 3 3
##  [7561] 2 2 4 4 1 1 1 1 4 3 2 1 3 1 2 3 1 3 1 3 1 1 3 1 2 4 3 4 2 4 4 3 2 4 3 2
##  [7597] 4 2 1 1 4 4 1 1 1 3 2 4 4 1 1 4 1 1 2 4 4 4 2 3 3 4 3 1 2 3 4 1 1 2 1 3
##  [7633] 1 1 4 3 3 3 3 4 3 4 1 2 4 1 4 1 1 2 3 2 1 2 4 3 4 3 1 1 1 4 4 4 3 2 2 3
##  [7669] 3 1 1 1 3 1 3 4 2 1 2 3 3 1 2 1 3 3 4 3 2 4 3 1 1 4 4 3 4 1 1 2 4 3 4 4
##  [7705] 2 2 4 4 3 4 2 4 2 3 2 1 1 3 3 2 3 2 3 3 3 3 1 3 3 1 1 1 3 4 1 1 1 4 3 1
##  [7741] 3 3 4 4 4 3 1 4 2 3 3 4 4 4 2 4 4 4 4 2 4 2 1 1 3 2 3 2 4 3 3 2 2 2 2 1
##  [7777] 3 4 4 2 4 1 4 3 3 1 2 1 4 1 2 3 3 2 4 4 3 4 2 3 4 3 1 4 2 3 1 4 2 4 1 4
##  [7813] 2 1 2 2 3 1 3 1 1 4 3 4 1 4 3 1 1 3 2 2 1 3 2 4 3 2 2 4 1 1 2 4 3 3 3 4
##  [7849] 3 1 1 4 4 1 3 2 1 4 2 3 4 1 1 2 3 1 4 3 2 4 2 1 3 3 3 2 2 2 2 3 3 2 3 2
##  [7885] 2 3 3 2 3 2 1 2 1 3 1 2 1 4 3 1 4 3 1 3 2 2 1 4 3 4 2 3 3 2 3 4 3 4 2 4
##  [7921] 3 4 3 1 2 1 1 1 2 1 3 4 3 2 4 1 2 1 4 4 1 2 4 2 3 3 2 3 1 3 3 1 2 1 1 1
##  [7957] 4 4 2 2 1 3 4 1 3 1 2 4 1 1 3 2 1 3 1 3 3 2 3 1 1 4 3 4 2 4 1 4 2 2 1 4
##  [7993] 4 4 3 1 4 2 3 1 2 2 4 2 4 2 1 4 4 3 3 3 4 3 1 1 3 2 3 3 3 1 3 3 1 2 2 3
##  [8029] 1 1 2 1 4 1 3 3 2 1 3 4 2 4 3 3 1 3 2 1 1 3 3 3 4 1 3 4 4 3 4 2 1 1 1 1
##  [8065] 2 4 1 3 2 1 1 3 4 1 3 1 3 4 3 1 2 3 1 3 1 3 4 3 4 3 4 3 1 2 1 3 4 3 3 2
##  [8101] 2 1 2 2 2 4 4 1 3 3 2 3 3 1 4 1 4 1 1 2 2 3 3 1 1 4 3 4 2 3 3 1 3 3 3 2
##  [8137] 3 3 4 4 3 3 4 3 1 4 1 2 1 2 1 1 3 3 2 4 1 2 2 2 2 2 4 4 2 4 4 1 1 2 1 3
##  [8173] 3 1 2 4 3 3 4 4 2 4 3 2 1 3 2 3 3 3 2 1 3 1 4 3 1 1 3 1 1 3 1 3 3 3 3 2
##  [8209] 4 4 4 4 1 4 2 1 4 2 3 4 3 3 2 3 1 1 1 1 4 3 4 3 1 1 4 2 4 2 1 4 1 1 2 3
##  [8245] 2 3 4 3 2 3 4 2 1 4 2 1 2 3 4 4 1 1 4 2 2 4 4 2 2 3 4 3 1 2 3 1 4 2 3 3
##  [8281] 2 4 4 3 1 4 1 1 3 2 2 4 4 2 2 1 3 2 3 3 3 3 3 4 2 4 1 4 2 3 1 3 1 4 2 1
##  [8317] 4 4 3 1 3 2 1 3 3 4 1 2 1 3 4 2 2 3 3 4 3 3 2 4 1 2 1 2 3 2 1 4 4 2 3 3
##  [8353] 3 3 1 1 3 1 2 3 4 4 2 4 3 3 1 4 2 2 1 3 3 1 3 2 1 3 2 4 2 1 1 1 3 1 4 1
##  [8389] 2 2 3 1 3 4 3 1 2 3 3 3 3 3 3 3 2 2 3 1 3 2 1 1 2 3 2 2 2 1 2 4 2 4 4 3
##  [8425] 2 2 4 3 4 3 1 4 4 4 1 1 4 1 4 4 1 4 4 2 2 2 1 3 4 4 3 3 1 1 3 3 4 1 1 1
##  [8461] 2 2 3 2 2 1 2 3 1 1 2 2 3 3 4 4 1 3 3 1 4 1 3 2 3 2 2 4 4 3 3 1 1 4 2 2
##  [8497] 2 2 4 1 4 2 4 4 1 3 1 2 1 1 3 3 2 3 2 4 4 3 1 3 4 3 3 2 3 4 3 3 4 3 4 4
##  [8533] 1 4 4 2 2 4 2 3 1 3 3 3 4 2 4 3 1 2 1 4 4 4 3 4 2 1 1 4 4 3 3 2 4 1 1 1
##  [8569] 1 4 2 1 2 3 4 1 2 1 2 4 2 1 1 4 3 1 2 4 1 2 4 1 1 1 2 3 2 1 3 1 1 3 2 2
##  [8605] 1 2 1 3 1 3 3 1 2 3 2 3 4 4 3 3 4 4 4 2 2 3 1 3 3 4 4 4 4 3 2 1 4 1 4 1
##  [8641] 1 3 2 4 1 2 2 3 4 2 1 1 4 1 2 1 2 1 1 3 3 3 4 3 3 4 3 4 4 4 2 2 1 1 2 3
##  [8677] 4 4 3 3 1 3 2 3 1 4 3 2 2 1 2 1 2 4 2 1 4 1 4 2 3 3 1 1 3 2 3 3 2 3 2 2
##  [8713] 4 3 2 1 3 1 1 2 4 3 4 3 3 1 3 3 1 4 3 2 4 2 4 2 1 3 4 2 3 2 1 1 4 2 3 4
##  [8749] 1 2 4 4 3 4 4 1 1 1 2 2 3 1 2 1 4 2 1 3 3 3 2 1 2 4 4 3 2 2 1 1 4 1 3 3
##  [8785] 2 4 3 3 3 1 2 3 3 3 2 3 3 1 2 3 2 4 3 3 3 4 4 1 3 1 1 1 1 1 3 1 3 1 3 2
##  [8821] 1 2 2 2 3 4 1 3 1 1 4 1 3 3 4 3 4 4 4 2 3 1 1 2 1 2 1 1 1 4 1 3 1 1 2 1
##  [8857] 2 1 2 4 1 2 1 1 1 1 3 2 4 3 3 1 1 2 3 3 1 4 2 1 4 4 1 2 2 4 3 4 3 3 2 1
##  [8893] 3 1 3 2 2 3 1 2 4 1 2 1 3 1 4 4 3 3 3 3 1 1 1 4 2 3 3 2 4 2 3 3 2 4 1 3
##  [8929] 2 2 1 2 1 4 1 2 3 4 1 4 2 1 4 1 3 1 3 1 2 1 3 4 3 1 4 3 4 1 4 1 4 3 4 4
##  [8965] 3 4 2 4 1 2 3 1 3 4 3 2 3 1 4 1 2 2 3 3 1 3 2 3 1 1 2 3 1 1 3 3 2 2 4 4
##  [9001] 2 4 3 2 1 2 4 3 1 3 2 3 4 3 1 1 3 1 3 4 3 3 1 1 3 3 3 1 1 3 3 1 4 2 3 3
##  [9037] 2 3 3 1 4 3 1 2 3 2 1 4 3 4 4 4 3 1 2 3 4 3 3 4 3 2 4 3 1 3 3 2 1 2 3 1
##  [9073] 1 3 2 1 1 4 2 3 3 3 3 2 1 3 4 3 2 1 2 1 2 3 2 1 4 2 1 3 2 1 4 3 4 1 2 2
##  [9109] 3 2 2 1 1 3 3 4 2 3 3 2 2 2 3 3 3 3 4 4 2 4 4 4 4 1 2 4 4 3 1 3 2 4 1 1
##  [9145] 4 3 3 1 2 2 2 2 3 1 1 4 4 2 3 2 2 3 2 4 1 4 1 2 1 4 2 2 4 3 4 1 1 4 1 4
##  [9181] 1 1 2 2 3 3 3 1 1 1 3 3 2 4 1 2 4 3 3 2 1 3 4 3 4 1 3 1 4 2 1 4 1 3 4 4
##  [9217] 4 4 1 2 3 3 3 1 3 2 3 4 1 1 1 3 3 4 2 1 1 4 2 2 4 3 1 1 3 4 2 3 4 1 3 4
##  [9253] 2 2 2 4 3 2 3 3 3 4 3 2 1 3 1 4 2 3 3 1 1 1 1 3 3 1 4 4 2 4 2 2 1 1 1 2
##  [9289] 4 1 2 4 4 3 1 2 3 2 1 3 1 3 2 2 4 2 2 4 3 4 2 3 4 3 1 4 1 2 3 3 1 3 3 4
##  [9325] 1 3 3 1 4 1 4 2 4 2 3 1 1 3 2 1 1 2 3 4 4 2 1 3 2 2 3 2 2 4 3 4 2 2 4 3
##  [9361] 1 2 3 3 4 4 3 3 4 4 4 2 2 3 1 2 4 3 4 4 2 4 4 1 2 2 2 3 1 3 4 1 4 4 1 4
##  [9397] 3 3 4 3 4 1 3 2 2 1 2 2 3 3 4 3 3 3 3 3 2 2 3 3 4 1 3 4 1 2 3 3 1 3 2 4
##  [9433] 2 3 2 2 1 3 4 3 4 3 3 1 4 1 2 1 3 4 3 1 3 3 3 3 3 1 4 2 4 1 4 3 1 1 4 1
##  [9469] 1 2 1 4 2 1 2 3 3 3 1 3 3 1 3 2 3 3 1 2 2 3 1 3 3 3 3 2 4 2 2 1 3 4 3 1
##  [9505] 1 1 3 2 4 3 2 4 3 1 1 3 2 1 2 3 2 2 4 1 3 4 2 1 3 4 1 3 4 3 2 2 3 1 3 4
##  [9541] 3 2 3 1 1 1 4 4 3 3 4 3 4 1 1 4 3 2 2 3 4 4 4 1 4 4 1 3 4 2 3 1 4 3 3 1
##  [9577] 4 3 1 2 3 3 1 1 2 2 3 1 4 2 1 2 3 4 1 4 2 4 3 3 4 3 1 1 4 4 3 1 3 2 4 1
##  [9613] 4 4 2 3 2 1 3 4 1 2 2 3 3 1 1 2 3 2 3 1 1 4 2 4 2 1 4 1 4 1 3 1 3 3 1 4
##  [9649] 1 3 3 3 4 1 3 3 1 1 2 3 4 3 1 2 2 3 1 3 3 1 3 1 3 4 3 4 4 3 4 2 1 3 1 3
##  [9685] 3 4 3 2 1 1 3 2 4 1 4 1 3 1 1 1 1 1 3 1 4 3 4 3 2 3 3 3 4 3 1 3 1 3 1 4
##  [9721] 4 1 1 1 3 4 1 3 3 1 1 1 3 4 2 2 2 2 4 3 3 2 3 3 3 3 3 1 3 4 3 1 3 1 3 1
##  [9757] 2 3 3 2 3 2 4 1 3 2 3 1 1 2 4 4 3 1 3 4 4 1 2 1 2 4 2 2 1 1 4 1 3 2 1 3
##  [9793] 2 3 3 2 1 4 4 2 1 1 3 3 1 2 3 4 3 1 4 2 3 4 3 3 3 1 4 4 2 1 3 1 3 2 2 3
##  [9829] 3 2 1 3 1 2 1 4 1 3 1 2 3 3 4 4 1 3 3 1 1 3 2 4 3 2 1 3 3 1 3 4 1 1 3 3
##  [9865] 3 4 4 2 2 3 3 4 1 1 2 3 1 1 1 3 1 2 1 2 1 3 3 4 2 3 4 3 1 3 1 4 2 2 2 2
##  [9901] 1 1 2 1 2 2 4 2 3 4 2 4 2 1 3 4 4 3 1 4 4 2 3 1 1 2 3 2 1 1 1 3 3 2 4 3
##  [9937] 1 1 2 4 3 4 4 3 3 1 4 2 1 4 3 2 4 3 3 2 1 4 3 1 3 3 4 4 2 4 4 3 4 4 1 4
##  [9973] 1 1 1 1 4 2 3 3 4 4 1 4 4 1 2 4 3 2 3 3 3 4 4 3 2 1 2 1 3 1 4 3 4 4 4 2
## [10009] 4 1 4 3 4 2 4 3 2 1 1 4 3 2 4 3 4 3 1 3 1 2 4 4 4 1 2 1 3 1 3 3 1 3 2 3
## [10045] 3 1 4 4 2 1 1 2 3 2 1 1 2 2 3 3 1 3 1 3 1 2 4 4 1 4 4 1 3 3 4 3 3 3 2 4
## [10081] 4 1 2 1 4 3 2 1 4 3 3 1 1 3 1 4 4 3 3 4 3 4 3 4 2 2 3 4 3 3 2 3 3 2 2 4
## [10117] 2 4 3 2 3 1 4 2 3 3 1 4 1 2 4 1 4 2 3 2 4 2 3 2 3 3 1 2 2 4 1 2 3 2 4 3
## [10153] 3 1 1 3 2 4 2 4 4 2 1 4 2 2 3 4 2 2 3 3 3 3 3 4 3 2 2 1 4 1 2 3 2 4 4 4
## [10189] 3 4 3 3 1 2 2 4 2 4 2 3 4 2 1 1 1 2 1 4 3 1 3 4 2 4 2 4 1 2 4 3 4 4 4 4
## [10225] 3 3 2 3 3 2 2 1 3 3 3 3 3 1 2 3 1 4 4 3 4 1 3 1 2 3 4 3 1 3 2 3 4 2 3 3
## [10261] 4 3 1 1 3 3 3 2 3 3 3 4 4 2 1 2 2 3 1 1 3 4 3 1 4 1 2 4 4 4 1 1 4 2 2 1
## [10297] 3 4 3 3 1 3 4 3 3 2 2 3 1 2 2 4 1 3 1 3 1 4 1 4 4 3 4 4 4 2 1 4 2 3 1 3
## [10333] 1 1 2 1 3 1 4 4 2 3 3 2 3 2 3 1 2 3 1 4 1 3 3 1 4 2 3 1 2 1 1 2 3 2 3 2
## [10369] 1 2 4 4 1 3 3 3 4 3 1 3 3 4 1 1 2 4 3 1 1 3 1 3 2 2 3 1 3 1 1 3 4 3 1 2
## [10405] 2 4 1 4 3 4 2 2 4 3 1 4 3 1 4 2 1 4 3 2 4 3 3 3 2 1 3 4 4 4 4 3 3 2 2 1
## [10441] 2 3 3 2 4 2 4 4 4 3 1 1 3 3 2 1 1 4 4 2 3 2 2 2 2 1 3 2 4 1 3 3 4 1 3 1
## [10477] 3 3 1 1 3 1 2 1 1 1 2 4 2 4 3 4 1 3 1 3 2 4 3 2 4 3 1 2 3 1 1 2 2 1 4 4
## [10513] 2 3 1 4 2 1 3 1 4 4 2 4 1 2 4 1 4 3 2 2 1 1 4 1 1 4 2 2 1 1 2 1 4 4 3 3
## [10549] 3 3 1 1 3 1 2 3 3 3 1 2 4 1 3 2 3 3 2 4 3 4 1 4 2 3 1 4 4 3 4 4 3 3 1 2
## [10585] 4 2 2 3 2 2 3 2 1 3 4 4 2 4 3 4 1 2 4 4 1 4 1 3 1 3 4 1 1 1 1 4 4 2 4 1
## [10621] 1 4 4 2 2 1 2 4 2 3 2 1 3 4 4 3 4 3 4 4 3 1 2 1 2 4 3 4 1 1 1 4 1 4 1 4
## [10657] 3 4 2 1 4 4 3 1 4 2 3 1 3 1 2 2 3 2 3 1 2 4 2 3 1 3 3 2 3 2 4 3 1 4 3 1
## [10693] 1 3 3 4 2 1 1 4 4 1 4 3 3 3 1 4 3 3 3 3 3 3 2 3 4 4 4 2 2 4 4 2 4 2 2 3
## [10729] 3 4 1 3 2 4 3 1 4 3 2 2 3 3 1 4 2 3 1 1 3 4 3 4 1 4 1 2 4 1 3 3 3 3 3 4
## [10765] 3 3 3 1 2 1 2 4 2 4 1 4 4 1 1 3 3 1 1 4 4 4 4 3 1 1 4 2 2 3 3 4 4 3 1 4
## [10801] 4 3 4 1 4 3 2 1 1 3 1 3 4 4 1 3 3 3 3 1 4 2 1 4 2 3 4 2 3 1 2 4 3 2 2 2
## [10837] 3 1 1 3 4 4 3 2 3 1 1 2 4 1 3 2 1 1 3 2 1 2 3 2 3 2 3 2 2 1 2 2 3 3 1 3
## [10873] 2 3 4 4 3 1 2 4 2 4 4 4 1 2 2 4 1 1 3 4 1 4 2 4 3 3 2 3 2 4 2 3 3 3 2 4
## [10909] 2 2 4 4 2 1 3 1 1 3 2 1 4 2 4 2 4 2 3 2 2 3 1 3 4 3 3 4 3 1 3 2 3 2 1 2
## [10945] 2 1 2 4 1 4 3 1 3 4 4 4 4 2 2 4 2 3 2 3 2 4 3 1 4 1 3 2 2 3 1 3 4 3 3 2
## [10981] 1 4 3 3 3 4 1 1 3 4 1 1 4 1 2 4 3 3 2 4 4 3 1 3 3 3 3 1 4 2 3 2 1 3 2 4
## [11017] 1 4 2 2 3 1 3 2 3 3 4 4 1 2 1 1 1 1 4 4 3 4 1 4 1 3 2 2 1 1 3 1 2 3 2 1
## [11053] 2 3 4 4 3 3 1 2 4 2 1 4 3 3 2 3 1 4 1 1 3 2 4 4 1 3 2 3 4 1 4 3 1 3 3 1
## [11089] 2 4 1 1 3 2 4 3 2 4 3 1 4 3 4 2 1 2 2 4 2 4 4 3 3 3 4 1 3 2 3 4 3 2 4 3
## [11125] 1 2 2 4 3 3 1 4 3 2 4 3 1 2 3 3 3 1 2 2 4 3 1 3 4 4 2 1 2 4 1 2 3 4 1 4
## [11161] 4 3 2 4 4 4 2 4 2 4 3 1 3 3 2 4 4 1 2 2 2 2 3 4 1 1 4 4 2 4 4 3 4 1 1 4
## [11197] 2 1 1 2 3 2 1 4 2 1 1 2 3 2 1 3 3 3 2 4 1 2 1 1 4 1 2 1 4 1 3 1 1 3 1 1
## [11233] 2 4 3 1 1 4 4 2 2 3 3 3 2 4 1 4 2 4 3 3 4 2 1 2 3 1 1 3 3 3 4 4 1 2 3 3
## [11269] 4 1 1 3 1 1 1 4 4 3 1 3 1 3 2 1 1 2 4 1 1 3 3 3 2 2 2 2 3 1 1 1 2 3 1 2
## [11305] 3 2 2 4 4 2 2 4 1 2 1 3 1 1 2 3 2 1 2 3 3 4 3 3 4 3 1 1 3 3 3 1 4 1 2 3
## [11341] 3 3 1 3 2 4 2 3 4 2 3 1 4 3 4 1 1 3 3 2 1 1 4 4 1 2 4 3 1 1 3 3 1 4 1 1
## [11377] 1 1 4 1 1 3 4 3 4 3 2 2 4 2 1 2 3 3 3 1 3 4 1 4 2 1 2 3 1 3 3 1 1 3 2 3
## [11413] 2 1 3 1 3 3 1 3 3 3 2 3 4 2 3 3 2 4 1 4 2 4 1 3 4 1 2 3 1 4 3 4 2 3 3 3
## [11449] 1 4 3 3 1 2 4 3 4 4 2 2 3 1 1 2 2 3 3 2 1 2 3 3 4 2 1 1 3 4 3 1 2 2 3 2
## [11485] 4 2 2 4 2 2 3 4 3 4 1 1 3 1 2 1 4 4 3 3 1 3 3 1 3 3 4 1 3 1 1 3 3 2 4 2
## [11521] 1 1 3 1 1 3 2 4 3 4 2 2 1 1 3 4 2 3 4 1 1 4 3 2 1 3 1 4 4 3 2 2 4 3 1 1
## [11557] 4 3 3 4 2 3 2 2 4 4 1 3 2 4 3 3 4 2 1 3 3 3 3 1 3 3 1 4 4 4 2 4 1 1 3 3
## [11593] 3 3 3 3 1 4 4 2 3 2 1 2 2 4 4 3 4 4 1 3 3 1 3 3 1 1 4 1 1 2 1 3 1 2 2 4
## [11629] 3 1 4 4 1 2 4 1 3 3 2 3 3 4 3 1 1 1 3 3 2 1 4 3 2 3 3 3 1 2 2 3 1 3 1 1
## [11665] 3 3 4 3 2 3 4 4 2 3 2 1 2 3 1 3 4 1 3 1 2 1 1 1 4 1 1 3 4 2 3 3 1 1 4 3
## [11701] 4 4 2 3 2 4 1 3 2 3 4 1 3 3 4 1 1 3 2 1 3 2 2 1 2 1 1 1 2 3 4 4 1 1 3 4
## [11737] 1 1 4 4 2 1 1 1 4 3 1 4 1 1 2 1 3 4 1 3 1 1 3 2 4 1 1 1 3 3 2 2 1 3 4 2
## [11773] 2 1 1 1 3 2 4 1 2 3 3 4 2 1 3 3 4 1 2 3 1 1 2 3 3 2 1 2 2 2 3 3 2 2 4 4
## [11809] 2 4 3 2 2 4 2 2 1 3 2 1 4 3 1 4 4 1 3 3 2 4 3 4 2 1 2 4 1 2 3 3 3 2 4 2
## [11845] 2 3 1 2 1 2 3 2 4 3 3 2 1 4 1 3 3 3 3 1 4 1 1 2 3 1 3 1 4 3 4 3 1 2 3 1
## [11881] 2 2 4 4 4 4 2 1 3 4 1 1 3 4 1 3 1 2 4 1 3 4 2 1 1 4 3 3 3 3 3 4 3 3 1 1
## [11917] 2 2 1 2 4 3 3 1 1 2 3 1 1 1 4 1 1 1 4 3 3 4 3 4 3 1 2 4 4 4 1 4 4 3 2 3
## [11953] 2 3 3 2 2 1 2 1 4 2 4 2 2 4 2 2 4 2 3 4 4 3 2 3 3 1 2 4 3 1 3 1 4 3 1 4
## [11989] 1 4 2 3 1 4 3 2 1 3 1 3 1 2 1 1 3 2 3 3 4 1 4 2 1 4 3 2 2 2 2 3 3 4 4 2
## [12025] 4 3 3 4 3 3 2 4 2 2 3 1 2 1 3 3 3 1 2 3 3 1 1 3 1 3 1 1 4 4 3 2 1 3 3 1
## [12061] 3 3 4 1 1 1 4 4 4 3 3 3 4 2 2 4 1 3 2 1 3 3 1 1 4 2 3 1 1 4 4 3 2 2 4 3
## [12097] 4 3 3 3 1 4 2 2 1 3 3 2 4 1 4 1 2 1 3 4 2 4 3 3 4 1 3 2 2 1 3 1 4 2 1 4
## [12133] 3 2 4 3 4 3 4 3 1 3 4 2 4 1 3 4 3 1 2 3 1 4 3 2 1 1 1 3 4 3 1 2 4 1 2 3
## [12169] 1 2 4 4 4 3 3 3 4 3 4 2 2 4 4 2 4 4 4 3 3 2 3 4 3 1 1 4 2 2 2 1 1 2 4 3
## [12205] 1 4 3 1 2 2 3 1 4 1 1 2 3 3 3 2 3 3 3 3 3 3 4 1 3 3 3 2 1 3 3 3 3 3 3 3
## [12241] 3 4 2 3 1 4 4 1 1 1 4 1 1 4 3 2 4 2 3 4 1 3 1 1 4 2 4 4 3 1 3 3 3 1 3 3
## [12277] 4 1 1 4 2 1 4 3 4 2 4 3 1 3 1 3 3 2 2 4 2 3 4 4 1 4 2 4 1 4 4 1 4 4 2 2
## [12313] 1 1 3 3 1 1 1 4 2 1 4 3 4 2 1 3 3 3 4 1 2 3 3 4 3 2 2 1 2 3 3 1 1 1 1 1
## [12349] 3 4 4 1 3 4 1 3 3 3 3 3 3 1 2 4 4 3 3 2 4 2 2 4 3 3 4 3 1 2 3 2 1 1 3 4
## [12385] 1 4 1 2 3 4 4 1 1 3 4 4 4 4 3 2 1 1 1 2 2 4 4 3 3 4 4 2 3 3 2 3 2 4 3 4
## [12421] 3 3 1 4 4 2 4 1 3 3 3 1 3 3 3 2 4 2 3 2 2 3 3 2 1 3 1 3 3 2 2 4 4 2 2 3
## [12457] 3 2 4 3 3 1 2 1 1 2 3 2 2 1 1 1 4 3 3 3 1 3 3 1 4 4 1 2 4 3 1 4 4 1 4 1
## [12493] 2 1 3 2 4 2 1 4 4 3 2 3 1 3 4 4 1 3 3 1 3 2 3 1 2 3 3 3 2 4 1 1 3 4 2 2
## [12529] 4 1 2 4 1 2 2 3 1 2 1 2 1 3 3 3 2 1 1 1 2 4 3 4 4 1 3 2 1 3 3 2 4 4 2 1
## [12565] 1 3 4 3 1 1 4 1 2 3 4 4 3 4 3 2 3 2 3 2 2 1 2 3 3 1 1 4 2 1 1 3 2 1 1 3
## [12601] 2 3 3 1 1 3 2 2 4 4 2 3 4 2 1 1 2 3 3 2 2 2 4 4 3 3 3 3 2 1 4 1 4 1 3 3
## [12637] 3 1 3 3 2 1 1 3 1 3 3 3 4 2 1 3 4 4 3 1 3 3 3 3 3 2 2 2 2 3 3 1 1 1 2 2
## [12673] 1 1 3 1 1 4 4 1 2 4 2 1 4 3 3 3 1 1 2 2 1 2 4 3 3 2 1 4 3 3 4 3 3 3 3 3
## [12709] 3 1 4 3 2 2 4 3 1 1 2 1 1 3 2 1 2 1 3 3 3 3 3 2 4 1 4 3 2 4 3 2 3 1 1 1
## [12745] 3 3 1 1 2 2 1 1 2 3 3 1 4 1 1 3 2 3 4 4 1 4 2 2 4 3 1 1 1 1 4 3 1 4 1 1
## [12781] 3 1 4 3 2 3 4 3 4 2 2 2 4 1 4 2 4 2 1 3 4 3 3 4 2 1 1 4 4 4 1 4 2 1 3 3
## [12817] 3 4 1 4 1 3 3 2 3 3 3 2 2 3 3 4 3 2 4 1 4 3 4 3 2 1 4 1 3 1 4 1 3 3 1 3
## [12853] 1 2 2 3 4 3 1 1 3 3 2 1 1 3 3 1 4 2 4 1 2 1 4 1 1 4 1 2 4 2 2 3 3 4 1 1
## [12889] 4 1 4 2 1 4 2 1 3 3 2 1 2 4 1 1 3 4 3 3 4 4 2 1 4 3 3 2 4 4 2 4 2 1 3 2
## [12925] 3 3 4 4 4 2 4 1 1 2 3 3 4 1 1 3 3 4 1 3 3 4 1 1 1 2 4 4 1 2 2 4 3 4 3 4
## [12961] 1 1 2 3 3 3 1 4 3 4 1 4 4 3 4 2 1 4 3 4 4 2 3 1 2 1 2 1 4 1 2 1 3 2 3 1
## [12997] 2 2 1 3 2 2 3 1 1 4 2 2 3 2 2 3 2 2 4 4 2 4 4 2 3 2 3 3 4 3 4 1 3 3 3 2
## [13033] 1 4 1 1 3 3 2 2 4 4 1 4 1 3 2 1 2 4 3 3 1 3 4 3 4 1 1 1 4 3 3 1 1 4 2 4
## [13069] 2 3 2 1 4 1 3 4 2 1 1 3 3 1 2 3 1 2 3 2 2 4 3 2 4 2 2 3 4 2 1 2 2 2 3 1
## [13105] 4 1 3 2 3 1 3 3 3 4 4 3 1 3 2 2 4 4 2 2 2 3 4 4 2 1 2 1 4 1 3 3 1 3 2 1
## [13141] 2 4 2 4 2 1 3 3 1 4 3 1 2 2 3 4 2 2 3 2 2 1 3 1 3 2 4 4 4 1 3 3 1 3 3 4
## [13177] 4 2 4 3 4 3 1 2 2 2 2 1 4 1 4 2 2 1 1 3 1 2 1 2 1 4 3 1 1 3 4 1 1 1 2 1
## [13213] 3 3 1 4 3 3 2 2 3 3 3 1 3 4 2 4 2 1 3 1 4 2 1 3 1 1 4 1 3 1 4 1 1 3 3 1
## [13249] 4 2 1 4 3 1 1 4 2 4 4 2 2 3 4 2 1 1 3 3 1 3 3 2 4 4 2 2 1 3 2 4 3 2 2 2
## [13285] 4 2 4 1 3 1 3 2 3 3 2 3 1 3 3 2 1 2 4 4 3 1 3 2 2 4 2 3 2 4 3 2 3 4 3 2
## [13321] 3 2 2 1 1 4 2 1 2 4 4 1 4 3 3 4 3 1 2 4 2 3 4 2 3 3 4 1 4 4 1 3 4 3 3 4
## [13357] 2 4 3 3 3 3 3 1 2 3 3 2 1 2 1 3 4 3 1 2 2 3 3 2 3 4 2 4 3 2 3 4 4 4 4 3
## [13393] 4 2 4 2 2 2 2 4 4 2 2 2 3 3 1 3 1 3 2 3 4 1 3 2 4 4 1 3 4 3 2 4 4 4 3 4
## [13429] 1 3 3 4 3 1 2 4 3 1 3 3 1 3 4 1 2 2 3 4 4 1 3 4 3 3 1 4 3 3 2 1 1 4 3 1
## [13465] 1 2 4 3 2 4 1 3 4 1 4 1 2 3 1 2 2 1 1 1 4 1 1 4 2 3 2 1 1 4 3 4 3 2 3 1
## [13501] 1 4 3 1 4 4 3 2 3 4 1 4 3 3 2 4 4 4 4 3 1 1 2 4 4 2 3 1 3 1 4 1 4 2 3 1
## [13537] 2 3 2 1 1 3 3 4 3 3 3 2 1 1 1 2 2 2 2 4 3 1 1 4 2 2 3 1 2 1 4 4 2 3 1 4
## [13573] 2 4 4 2 3 1 3 1 3 1 2 3 4 4 3 4 2 4 3 4 1 4 2 4 2 3 4 4 1 2 1 3 1 3 4 2
## [13609] 4 4 3 3 2 3 4 1 1 4 2 4 4 1 3 1 1 1 1 4 1 2 1 3 3 3 1 4 1 2 3 4 2 1 4 1
## [13645] 3 4 3 3 2 4 4 1 2 3 3 1 3 3 3 1 1 1 3 1 3 4 3 4 2 1 2 2 1 4 2 2 4 1 1 3
## [13681] 2 1 3 1 4 2 1 1 3 1 4 4 4 3 1 3 4 2 1 3 3 1 4 2 3 1 3 1 1 3 2 2 2 3 1 3
## [13717] 4 3 3 4 2 2 2 2 1 1 2 3 4 1 1 4 3 3 3 3 3 2 1 2 4 4 3 1 2 3 1 2 3 1 3 2
## [13753] 1 4 2 3 1 2 3 3 3 2 1 3 3 4 3 4 2 1 2 4 3 2 3 3 3 2 4 4 1 2 3 3 1 4 2 4
## [13789] 3 3 4 1 2 4 4 3 3 2 4 2 4 1 4 1 3 1 4 1 3 1 4 3 3 3 4 1 2 3 2 4 2 3 4 4
## [13825] 1 2 2 4 1 4 1 1 3 4 3 1 1 2 1 4 2 3 4 1 4 2 1 2 1 2 3 1 4 1 4 1 3 2 1 3
## [13861] 1 3 3 4 2 3 2 1 1 3 2 3 2 3 3 4 2 4 1 3 3 2 2 3 3 3 4 3 3 3 1 3 3 1 1 2
## [13897] 4 1 2 4 4 4 3 2 4 2 2 4 3 4 2 3 1 4 3 3 3 2 1 4 2 3 4 4 4 3 3 1 4 2 3 3
## [13933] 4 4 1 2 1 1 2 1 1 1 4 3 4 1 1 1 2 4 1 3 1 3 3 3 4 3 2 1 3 3 2 3 4 3 4 3
## [13969] 3 1 1 2 3 2 4 1 3 2 4 3 2 1 1 1 2 4 4 2 1 3 4 1 1 2 1 2 4 1 1 3 4 3 3 3
## [14005] 2 4 3 3 3 4 4 3 2 3 2 1 4 1 3 4 2 1 3 4 1 4 2 4 3 2 1 4 2 3 3 2 3 4 1 1
## [14041] 2 1 3 3 3 4 2 2 1 4 4 4 1 1 1 2 3 3 4 2 3 3 1 1 3 4 2 1 4 4 1 4 4 3 3 2
## [14077] 3 2 1 1 4 4 3 1 4 4 2 3 1 2 3 1 2 2 4 4 4 2 4 2 3 3 3 2 1 2 3 2 3 4 1 1
## [14113] 1 2 4 3 1 1 4 3 4 3 3 4 4 3 1 1 3 4 2 1 3 1 3 3 1 4 3 4 4 3 4 2 1 3 1 4
## [14149] 4 2 3 4 1 4 1 2 3 1 3 3 4 4 4 3 3 2 4 3 1 4 1 3 2 3 1 4 1 1 4 4 2 2 2 2
## [14185] 3 1 2 4 3 4 1 1 3 4 4 1 4 3 2 2 3 2 2 1 1 1 2 2 2 4 4 3 4 3 1 3 3 2 2 2
## [14221] 2 2 1 4 2 2 2 1 4 2 1 3 4 1 3 2 3 4 3 1 1 2 1 3 1 4 3 4 2 1 4 1 1 3 3 3
## [14257] 4 2 2 2 4 4 3 4 3 1 4 4 2 1 1 4 4 1 2 1 3 2 1 3 4 4 4 4 4 3 2 3 3 3 2 3
## [14293] 1 4 3 2 4 2 1 2 4 1 3 3 3 4 4 1 4 4 1 1 3 2 3 3 2 4 3 2 4 1 2 3 3 1 3 2
## [14329] 3 3 1 1 3 1 4 1 3 4 4 4 1 3 1 1 4 2 1 3 3 2 4 1 1 1 4 3 2 4 3 1 1 1 1 3
## [14365] 3 2 3 3 3 1 3 1 4 2 4 4 4 3 3 1 1 3 2 4 3 4 4 3 3 1 1 4 4 4 4 1 4 3 1 4
## [14401] 4 4 2 3 2 4 1 1 1 4 1 3 3 4 1 2 1 1 4 4 3 3 2 3 3 1 1 1 2 1 2 3 1 3 2 4
## [14437] 1 4 4 1 2 4 2 3 2 1 2 3 2 4 1 1 2 3 1 2 4 2 4 2 2 2 3 2 1 3 1 1 1 3 1 3
## [14473] 4 2 2 4 1 4 4 4 1 3 3 1 1 1 4 4 1 1 2 4 3 4 1 4 1 3 4 2 2 3 3 1 3 1 2 4
## [14509] 3 3 2 3 3 1 4 1 3 2 3 3 2 1 2 4 2 3 2 1 2 4 3 3 1 2 1 1 4 2 4 1 4 3 4 1
## [14545] 3 3 4 4 2 3 3 2 4 3 1 4 3 3 1 4 2 1 4 4 3 4 4 3 2 3 4 3 4 3 2 3 1 4 4 1
## [14581] 1 1 4 2 2 3 2 3 2 3 4 2 2 3 1 3 3 4 4 1 1 1 3 2 1 4 3 2 1 1 4 4 1 3 1 3
## [14617] 4 3 2 3 1 4 4 1 3 2 1 1 4 2 2 3 3 1 4 3 3 2 4 4 3 1 3 2 1 2 4 2 2 2 3 2
## [14653] 3 2 1 3 4 2 3 2 2 2 4 2 2 2 1 3 1 2 3 1 3 3 3 2 2 2 1 2 1 4 3 2 2 1 1 3
## [14689] 2 3 2 1 4 4 3 3 1 1 3 1 3 1 3 2 4 3 1 2 4 2 1 2 3 3 1 1 2 2 3 2 1 1 4 2
## [14725] 2 4 1 2 1 3 3 4 2 3 1 3 3 4 1 2 3 3 2 3 2 2 3 1 1 3 2 4 4 3 1 3 1 3 4 1
## [14761] 1 2 2 3 3 4 3 1 2 3 4 2 4 3 4 4 1 1 1 3 4 1 3 4 1 4 1 1 3 2 2 1 4 3 4 2
## [14797] 1 4 3 2 3 2 1 2 3 1 3 3 3 2 4 1 4 4 2 2 2 3 2 2 4 2 2 2 1 2 3 3 3 2 2 2
## [14833] 3 3 2 4 3 2 2 1 4 2 1 4 2 3 1 2 1 2 3 3 1 1 3 4 1 1 4 4 1 4 4 2 2 1 2 1
## [14869] 4 2 1 4 1 3 3 1 2 1 4 1 1 3 3 2 1 3 3 3 4 3 2 2 2 4 4 1 3 1 1 1 4 4 1 4
## [14905] 4 1 3 2 1 2 2 4 4 3 2 2 3 4 3 1 3 4 1 2 1 4 3 3 4 4 2 1 2 3 2 2 2 3 1 2
## [14941] 1 2 1 1 1 2 4 2 3 4 1 3 3 1 1 4 2 2 1 2 4 1 3 4 1 3 3 3 4 2 1 1 3 1 3 3
## [14977] 3 2 3 4 4 1 4 3 2 3 3 2 3 3 3 1 2 3 2 4 2 2 3 2 1 2 3 1 2 1 1 1 1 3 3 2
## [15013] 2 2 4 4 4 4 2 3 2 3 4 2 3 4 2 1 1 3 2 2 2 3 1 1 3 1 2 3 3 4 1 3 2 1 3 1
## [15049] 2 4 2 2 2 1 4 1 2 3 3 2 4 3 3 3 2 4 3 3 3 3 3 1 1 1 4 1 4 2 3 3 2 4 3 3
## [15085] 4 3 1 3 3 1 4 1 3 1 3 2 3 1 1 1 4 2 1 3 4 3 1 3 4 2 1 2 1 4 4 2 4 2 3 2
## [15121] 4 3 2 1 1 3 4 1 1 2 3 4 3 4 4 3 4 3 2 3 2 3 3 4 4 3 1 4 3 1 3 1 1 3 3 4
## [15157] 1 3 2 3 2 3 2 3 4 3 4 1 4 4 1 2 2 2 4 2 3 3 3 1 1 3 4 3 2 4 1 1 3 2 1 4
## [15193] 3 1 2 4 2 3 4 3 2 3 4 1 4 2 3 3 4 4 2 1 4 3 2 1 1 3 1 2 2 4 2 3 2 4 3 4
## [15229] 2 1 1 2 4 4 3 2 3 3 2 4 1 3 2 1 2 3 3 1 4 1 4 2 4 3 4 4 2 3 3 2 3 3 3 1
## [15265] 1 3 3 1 4 2 4 1 2 1 1 1 2 4 4 4 4 3 3 2 3 2 2 3 3 1 3 3 3 2 3 4 1 3 1 4
## [15301] 1 4 4 2 3 1 2 1 1 1 1 4 2 2 4 3 4 3 1 4 2 2 2 3 1 4 1 1 2 4 3 3 3 3 1 1
## [15337] 2 1 1 2 4 4 1 3 3 2 3 4 1 3 2 2 2 2 4 4 3 3 4 1 3 3 4 2 3 1 3 2 2 3 2 3
## [15373] 2 2 1 3 1 1 3 4 1 4 1 2 4 1 1 3 4 4 4 1 3 1 2 3 3 3 1 1 4 4 1 3 2 3 4 2
## [15409] 4 1 3 4 3 2 1 1 2 1 4 1 3 1 2 1 3 3 1 3 1 4 3 3 4 1 4 3 1 4 1 3 1 3 1 1
## [15445] 1 2 3 2 4 4 4 1 1 2 1 4 2 2 1 2 1 1 3 1 4 1 3 3 1 3 3 1 2 4 2 4 3 3 3 4
## [15481] 3 2 1 3 3 4 2 4 2 2 2 3 3 2 4 4 2 3 1 4 4 2 2 3 3 1 4 1 2 1 3 2 2 2 4 1
## [15517] 2 2 1 2 1 4 3 2 4 1 4 4 4 2 4 1 2 1 3 1 2 3 1 3 2 1 2 1 2 3 3 3 1 2 2 2
## [15553] 1 4 4 2 3 4 3 4 3 2 3 3 1 1 1 3 1 3 2 4 4 3 3 4 3 1 3 4 1 1 4 1 3 1 4 4
## [15589] 1 4 2 2 1 2 2 1 4 3 3 4 1 1 3 1 4 2 3 3 3 1 4 2 3 1 2 2 3 2 2 1 4 3 2 1
## [15625] 3 3 4 3 3 1 3 4 1 4 4 1 3 3 4 3 1 1 4 4 3 3 3 3 4 3 4 3 2 3 1 2 2 1 1 2
## [15661] 1 2 4 4 1 4 4 4 1 1 1 3 1 1 2 3 1 2 4 4 2 2 4 3 3 4 3 1 1 3 1 1 3 2 1 1
## [15697] 4 1 2 1 1 1 4 1 4 1 1 4 4 3 4 3 3 4 1 2 3 3 2 4 4 4 3 2 3 3 3 3 2 3 2 2
## [15733] 4 1 3 1 1 3 1 3 3 2 1 4 4 1 3 4 1 3 1 4 4 2 3 3 4 1 4 2 3 1 1 3 1 4 2 1
## [15769] 1 3 1 4 3 3 3 2 2 4 2 4 4 1 4 2 1 2 3 2 1 3 3 1 3 4 3 1 2 3 3 4 3 3 2 2
## [15805] 3 1 1 1 1 1 1 4 1 1 4 1 4 3 3 2 2 3 4 4 3 1 4 2 1 3 4 1 3 1 4 1 2 2 2 2
## [15841] 3 2 3 2 1 4 2 2 3 4 3 4 3 3 3 4 2 3 1 4 2 3 2 1 3 3 4 4 1 4 1 1 1 4 2 2
## [15877] 1 2 1 4 1 3 1 1 1 3 3 1 2 4 1 1 1 2 3 4 4 4 4 1 4 3 3 4 2 1 3 4 2 3 4 3
## [15913] 1 3 1 4 2 3 2 4 4 1 1 1 2 1 3 1 2 3 4 3 2 3 2 1 3 2 4 3 2 3 3 4 4 4 3 3
## [15949] 4 3 1 3 3 4 2 3 3 1 1 3 4 3 1 4 1 2 4 2 3 3 1 3 1 4 1 2 1 1 4 1 3 3 3 3
## [15985] 3 1 4 1 4 1 3 2 1 3 4 3 3 3 1 2 4 2 1 1 3 1 3 1 4 3 4 3 1 4 3 3 1 1 3 3
## [16021] 2 1 3 1 3 1 4 1 4 3 2 1 1 4 3 3 1 3 3 2 1 3 1 1 3 2 1 3 2 3 1 2 3 1 1 3
## [16057] 4 2 2 3 4 2 1 3 3 1 1 3 4 2 3 4 1 4 1 3 4 2 1 3 4 3 4 2 3 3 3 1 1 2 4 3
## [16093] 2 1 3 1 3 3 2 2 3 3 4 3 2 3 2 4 1 3 3 3 2 2 3 3 3 1 4 4 1 4 4 1 2 3 3 2
## [16129] 1 4 1 2 1 3 2 4 1 1 2 3 3 2 2 4 3 1 4 3 3 4 3 4 3 1 2 3 1 1 1 4 2 2 1 4
## [16165] 3 3 1 1 1 1 3 4 1 1 2 2 4 3 1 1 4 1 2 4 3 2 3 4 2 4 3 1 4 4 3 3 3 1 2 1
## [16201] 2 1 3 1 3 3 1 2 1 2 3 2 4 3 2 4 1 1 2 2 1 3 4 1 3 3 3 3 4 2 1 1 1 2 3 4
## [16237] 4 2 4 1 1 3 1 1 2 4 1 3 4 2 2 1 1 3 3 4 4 1 1 4 1 2 1 4 1 3 2 2 4 1 4 3
## [16273] 3 1 2 4 3 2 3 2 4 1 3 4 3 1 1 2 1 4 1 3 4 4 4 3 4 3 3 4 2 1 1 3 3 1 1 1
## [16309] 2 1 4 2 1 2 1 3 1 1 2 1 1 1 2 4 3 3 4 4 1 4 1 4 4 3 4 2 4 2 3 1 3 3 4 4
## [16345] 2 1 4 4 4 4 3 1 4 1 2 2 4 1 3 3 2 2 3 1 2 2 1 3 4 4 2 3 2 1 1 3 3 2 3 2
## [16381] 2 4 2 1 1 3 2 1 3 1 3 3 2 2 1 3 3 3 3 2 1 3 4 3 3 1 2 4 1 3 3 3 2 1 2 2
## [16417] 3 3 4 3 1 2 2 3 3 2 2 4 4 1 2 3 4 2 2 2 1 4 2 4 4 1 3 4 3 2 3 2 1 3 3 4
## [16453] 3 3 4 2 1 4 1 2 1 3 4 3 1 3 4 3 1 3 2 2 3 1 1 2 2 3 3 1 4 1 4 2 2 2 4 1
## [16489] 4 3 1 2 2 2 3 4 2 4 1 3 4 2 1 4 2 1 3 3 1 4 3 1 4 3 2 1 3 2 1 3 2 3 3 4
## [16525] 4 2 1 1 2 4 3 2 2 4 2 4 2 2 1 3 3 4 3 3 4 1 3 2 1 1 1 3 4 1 4 1 3 3 1 2
## [16561] 3 1 3 1 3 3 3 4 2 4 3 4 4 4 4 4 3 1 1 4 3 3 4 3 2 2 3 1 4 4 2 4 4 1 2 3
## [16597] 2 2 4 2 3 4 3 3 4 4 3 3 4 3 2 3 4 1 4 4 3 4 3 3 1 2 1 1 4 3 1 1 2 2 3 4
## [16633] 1 1 4 3 3 3 4 2 1 1 3 4 3 4 1 3 1 2 3 3 1 3 4 3 3 3 3 2 1 1 1 2 2 4 2 2
## [16669] 4 2 1 3 3 3 4 1 3 2 1 3 3 4 4 2 4 1 1 3 3 3 4 4 2 3 2 1 3 4 3 3 4 3 2 4
## [16705] 2 3 1 3 2 3 1 4 2 4 2 1 3 4 1 1 2 2 3 4 1 3 3 4 2 4 4 3 1 3 4 1 3 3 4 1
## [16741] 2 2 2 2 2 4 4 1 1 1 2 2 3 1 4 1 1 3 3 3 3 1 1 1 3 3 1 2 1 2 2 3 1 3 2 1
## [16777] 1 4 1 1 2 1 2 1 1 2 4 3 1 2 1 4 4 4 3 1 1 1 4 1 2 3 2 4 1 3 3 3 3 1 3 1
## [16813] 3 4 1 2 1 1 2 3 4 1 2 3 1 1 3 1 4 4 4 2 1 4 1 1 1 2 2 3 3 4 4 1 2 2 2 4
## [16849] 1 1 1 3 3 3 4 4 4 3 4 4 1 4 3 3 4 4 3 1 3 1 1 4 1 4 1 4 3 1 1 4 3 3 2 3
## [16885] 4 4 3 2 4 3 3 3 3 4 1 1 4 4 4 4 2 4 2 4 3 1 1 2 3 4 3 1 1 4 4 3 3 4 1 4
## [16921] 1 1 1 1 1 2 4 4 1 4 2 2 1 4 4 3 4 2 1 1 1 1 2 3 3 3 1 4 4 2 2 3 1 3 3 3
## [16957] 4 1 2 4 2 2 4 4 3 3 4 4 2 3 1 1 3 2 4 3 4 3 3 2 2 2 3 1 2 1 1 4 2 3 1 2
## [16993] 4 2 4 2 2 4 3 3 1 4 3 4 3 3 3 3 4 3 2 4 3 4 4 3 3 3 1 1 3 1 4 3 3 4 2 3
## [17029] 2 1 3 4 3 1 2 4 1 2 3 1 1 3 3 3 2 1 4 2 1 4 4 2 4 3 1 1 2 2 2 3 4 3 3 1
## [17065] 2 4 1 1 1 3 3 2 3 4 1 3 2 2 1 4 2 1 3 1 3 1 2 3 4 2 1 2 3 2 4 1 3 4 3 3
## [17101] 4 1 4 4 3 3 4 3 2 4 3 1 1 1 3 2 3 1 2 4 4 4 1 1 1 2 3 1 1 2 2 2 1 1 1 3
## [17137] 4 3 2 2 3 1 3 3 4 3 4 4 2 4 2 1 3 1 1 1 3 2 1 1 2 4 3 3 2 3 3 3 4 3 2 1
## [17173] 1 1 3 2 2 3 3 2 2 4 1 4 2 2 2 1 3 2 3 3 2 4 2 1 2 1 1 1 4 1 1 1 2 3 3 3
## [17209] 3 2 4 1 2 4 3 2 1 4 3 2 3 4 1 3 2 1 1 2 2 4 1 3 2 2 4 3 3 1 2 1 3 3 3 2
## [17245] 1 2 4 1 3 2 1 2 4 1 4 2 4 2 4 3 4 1 2 2 3 1 4 3 2 3 1 3 2 1 3 3 3 1 1 3
## [17281] 2 2 1 1 3 4 2 4 4 1 4 3 2 3 4 3 1 4 2 4 3 2 2 3 2 4 3 4 4 1 1 1 1 3 2 3
## [17317] 2 1 1 3 3 3 4 3 2 3 3 3 1 1 1 3 2 1 3 3 4 4 3 2 2 2 2 4 1 4 1 3 3 1 4 2
## [17353] 4 1 1 2 1 4 2 1 3 3 1 4 4 4 4 2 4 3 2 4 3 2 3 3 2 1 2 2 1 2 2 1 3 4 2 1
## [17389] 4 3 4 1 3 3 1 3 1 3 3 1 3 4 4 3 2 2 3 4 4 3 2 1 2 2 1 3 3 2 1 1 1 2 2 3
## [17425] 2 3 1 3 4 3 1 3 4 1 4 4 4 3 2 4 3 1 4 3 3 2 3 2 1 3 2 4 1 4 4 3 1 1 3 1
## [17461] 4 3 3 3 2 4 3 3 1 1 2 1 2 1 2 3 2 2 1 1 2 1 4 1 4 4 1 2 2 1 4 1 3 1 1 4
## [17497] 1 3 3 2 3 1 4 3 2 2 1 1 1 2 2 2 4 3 3 4 3 3 4 4 3 4 3 2 2 3 2 1 3 3 2 3
## [17533] 2 3 4 4 1 4 4 2 2 3 1 1 4 1 2 1 3 1 3 4 2 2 2 2 4 4 2 4 4 3 2 2 4 4 4 1
## [17569] 4 2 3 3 2 1 1 2 3 3 4 3 4 4 4 1 2 2 3 2 1 4 3 3 3 3 4 2 1 3 3 2 2 1 1 3
## [17605] 4 4 1 1 4 4 3 1 2 3 2 4 4 3 3 3 1 1 3 2 4 1 1 2 4 3 1 1 3 1 2 3 1 3 1 4
## [17641] 2 3 1 3 2 1 3 2 4 3 4 3 3 2 4 1 1 1 3 3 3 1 3 3 1 1 2 3 4 3 1 1 3 1 4 2
## [17677] 4 2 4 1 3 1 4 4 2 1 3 4 4 2 1 2 2 1 1 3 3 4 3 2 2 2 3 1 1 2 4 3 2 3 2 2
## [17713] 3 3 3 1 2 3 3 3 4 3 3 1 4 2 1 1 2 1 1 1 4 4 3 1 4 1 2 4 1 3 3 3 4 2 3 1
## [17749] 1 4 4 3 3 3 2 1 4 2 2 2 4 3 4 2 4 4 2 3 4 1 4 2 1 4 3 2 2 3 4 1 3 2 1 2
## [17785] 2 4 1 1 1 4 3 3 1 4 4 3 2 1 2 4 3 4 1 2 3 3 1 1 4 4 2 4 2 3 3 1 4 2 2 2
## [17821] 3 4 1 1 3 4 3 3 4 4 1 1 2 3 1 4 3 4 1 4 3 1 4 3 3 3 3 1 2 2 4 4 4 2 3 3
## [17857] 2 3 4 2 2 1 4 3 4 1 4 2 4 2 3 4 1 1 4 3 1 4 2 3 4 4 3 3 2 4 1 3 1 3 4 3
## [17893] 3 4 3 1 3 1 2 4 2 4 1 1 1 3 4 3 1 1 3 4 2 4 3 4 4 1 1 3 4 4 4 2 3 3 1 1
## [17929] 2 1 3 1 3 1 4 3 2 3 3 3 1 2 4 4 1 2 1 3 2 3 3 4 1 3 3 1 3 1 2 4 3 1 3 4
## [17965] 2 4 2 1 1 4 2 3 4 1 1 1 3 1 4 4 1 3 4 3 4 1 4 2 4 4 1 2 3 4 3 1 3 2 2 2
## [18001] 1 2 4 3 1 4 1 1 3 3 4 3 3 2 3 3 3 1 1 2 4 4 4 2 4 3 4 1 2 3 2 3 3 1 2 1
## [18037] 3 4 2 1 2 4 2 3 2 2 1 2 2 4 3 3 2 1 2 2 4 3 3 3 3 1 1 3 4 1 2 3 4 2 1 1
## [18073] 1 1 3 2 4 1 4 2 3 1 3 4 2 3 4 4 2 3 4 1 1 4 3 3 2 4 2 1 4 1 3 3 2 3 1 4
## [18109] 4 1 3 4 4 2 1 3 1 3 4 3 3 2 4 1 3 3 3 4 4 1 3 3 2 3 3 4 3 4 1 3 1 1 3 2
## [18145] 1 2 1 3 2 2 1 2 1 2 1 3 3 3 4 1 2 1 3 2 1 2 3 2 4 2 3 2 2 3 2 3 1 1 1 2
## [18181] 4 4 3 3 3 2 2 1 3 1 1 2 1 4 3 1 4 4 1 2 1 4 2 3 2 4 4 2 3 3 4 1 4 4 1 1
## [18217] 2 2 4 3 2 4 1 4 3 2 3 2 2 1 3 2 3 1 1 3 1 1 4 4 2 3 4 3 1 3 3 3 3 3 3 1
## [18253] 1 2 2 2 3 2 3 2 4 2 1 1 1 1 2 3 1 2 3 2 2 2 1 3 2 3 4 2 4 1 1 4 3 4 2 2
## [18289] 1 2 3 4 2 3 1 3 4 4 4 1 4 1 3 2 4 3 3 2 3 1 2 1 3 3 1 1 1 4 3 4 1 1 2 1
## [18325] 3 2 3 2 3 4 1 2 1 3 1 1 2 3 1 4 3 3 3 3 1 3 1 4 3 1 4 1 1 3 4 4 3 4 3 2
## [18361] 2 1 2 4 2 1 2 3 2 2 2 4 3 1 1 1 1 3 2 3 2 2 3 3 4 3 4 3 3 3 3 2 3 3 3 4
## [18397] 4 3 1 4 4 2 2 3 1 2 3 3 3 4 2 1 2 2 2 4 4 2 4 1 4 1 4 4 4 1 4 3 2 3 4 3
## [18433] 1 1 3 4 3 4 1 1 4 4 3 4 3 1 1 4 2 4 1 2 2 3 4 3 1 3 1 3 3 3 2 3 3 2 1 2
## [18469] 1 3 3 2 3 4 3 3 4 2 3 4 4 3 3 2 2 4 4 2 1 3 1 3 3 3 1 3 2 1 2 3 1 1 2 1
## [18505] 1 2 3 2 4 1 4 2 2 3 3 2 3 3 2 3 2 4 1 4 2 3 3 4 1 4 3 4 1 4 4 2 1 3 3 4
## [18541] 3 4 2 2 3 1 3 3 1 3 4 3 3 1 2 4 1 4 3 3 1 3 4 3 3 2 2 3 4 1 1 4 3 3 2 4
## [18577] 2 1 1 2 4 3 3 2 3 1 3 2 2 2 1 1 2 4 4 3 3 3 3 2 1 3 1 4 4 2 3 2 1 3 4 4
## [18613] 3 1 3 2 4 1 1 4 3 1 3 3 2 2 3 3 3 4 3 2 3 3 4 2 4 1 3 2 3 1 1 3 2 2 1 2
## [18649] 2 2 3 4 2 3 3 3 3 3 2 2 3 1 3 2 1 1 1 1 2 4 4 1 3 3 4 3 2 4 1 4 3 3 3 4
## [18685] 2 2 3 3 3 3 2 3 3 3 1 3 2 3 1 3 1 2 2 1 3 2 1 1 4 2 1 3 3 2 2 2 1 1 3 2
## [18721] 2 1 2 3 4 2 4 3 3 3 3 3 3 1 2 1 4 1 4 2 3 3 4 1 3 1 4 1 1 3 1 3 1 3 4 3
## [18757] 4 1 4 2 3 1 4 1 1 1 3 1 3 2 1 4 3 4 1 4 4 2 4 3 4 4 2 4 1 2 3 3 4 3 1 3
## [18793] 1 4 2 4 2 4 4 4 2 4 2 2 2 4 3 3 2 4 3 4 4 3 3 3 4 4 3 4 2 1 4 1 4 2 1 4
## [18829] 2 2 3 4 1 1 1 3 4 1 1 2 2 3 3 1 4 2 2 3 3 3 2 2 2 2 2 3 4 3 1 3 2 3 1 4
## [18865] 4 4 3 3 4 3 2 3 1 2 2 1 3 1 1 1 2 4 1 1 3 1 3 1 4 4 3 2 1 3 3 2 2 4 4 1
## [18901] 2 1 4 1 1 2 3 2 1 2 1 2 3 2 4 1 3 2 1 4 3 4 3 2 2 1 4 1 3 1 4 1 4 4 3 4
## [18937] 4 3 1 1 4 4 4 1 3 3 2 1 4 4 1 2 1 3 3 1 1 1 3 4 1 4 2 3 3 3 1 3 2 2 4 4
## [18973] 3 2 2 4 3 3 3 1 2 1 3 3 3 1 4 4 4 1 3 1 3 3 2 4 1 3 2 2 2 3 1 3 4 2 3 3
## [19009] 4 3 4 2 2 1 4 1 4 1 3 2 3 3 3 1 4 1 3 3 4 1 3 4 3 4 3 2 3 4 3 1 3 2 4 1
## [19045] 4 1 1 4 3 2 4 2 1 1 3 4 1 2 4 3 3 3 1 4 1 3 4 2 2 1 3 1 2 3 3 3 3 4 3 4
## [19081] 1 4 4 3 2 1 3 4 2 3 2 1 3 4 2 2 3 3 1 3 2 4 3 1 3 1 4 2 4 2 1 3 2 3 4 3
## [19117] 4 3 3 2 4 4 3 3 4 4 2 3 2 3 3 3 1 3 2 1 1 3 3 3 1 2 2 1 1 3 3 4 2 1 3 4
## [19153] 2 4 3 1 4 3 4 4 3 2 2 2 2 4 2 3 4 4 1 4 1 3 3 2 3 4 4 3 3 2 1 4 2 3 3 4
## [19189] 1 4 2 3 4 2 4 1 1 4 4 3 4 1 3 1 1 4 1 4 3 2 2 4 2 3 1 3 3 1 3 3 4 4 1 1
## [19225] 4 1 4 2 2 3 2 4 2 3 4 3 2 4 4 3 2 2 1 4 2 2 4 2 3 4 1 2 1 2 3 4 1 1 3 3
## [19261] 4 3 4 3 4 3 4 1 2 1 1 2 1 2 1 3 2 2 3 3 3 4 2 2 2 2 4 4 1 4 4 3 1 1 3 1
## [19297] 2 2 4 3 4 2 3 3 2 2 1 2 4 3 1 2 1 1 3 4 2 2 4 1 3 1 4 4 2 2 3 3 3 3 4 2
## [19333] 4 2 1 4 4 2 1 2 3 4 1 3 1 1 1 3 4 2 2 2 2 3 3 2 1 1 1 3 2 2 2 4 4 3 1 1
## [19369] 3 3 2 4 3 4 3 3 3 3 1 3 2 3 1 3 3 3 4 1 3 2 2 1 1 4 1 2 3 1 1 2 3 3 1 1
## [19405] 2 2 4 2 4 2 3 4 3 2 2 1 2 4 3 1 1 1 1 4 3 2 2 2 2 3 1 3 2 3 4 3 3 1 3 4
## [19441] 1 4 3 3 2 3 2 2 4 4 1 4 3 4 1 1 3 1 2 3 1 3 1 2 4 3 4 3 1 4 1 3 3 2 1 4
## [19477] 4 3 1 4 2 2 3 3 3 2 3 1 1 4 1 3 3 3 3 3 3 1 4 3 3 2 1 4 4 2 3 1 1 4 1 4
## [19513] 4 2 4 1 3 3 1 4 3 2 2 3 2 1 1 4 2 1 1 4 3 2 4 4 3 1 3 3 1 1 4 1 4 4 3 2
## [19549] 1 2 4 1 1 2 3 1 1 4 4 2 4 3 3 2 4 3 3 2 1 4 3 1 2 1 3 4 3 3 2 3 2 2 2 1
## [19585] 3 3 2 3 4 2 1 3 2 3 1 4 3 4 3 1 1 4 3 4 3 1 3 2 4 1 1 1 3 2 1 1 4 2 3 1
## [19621] 2 4 4 1 1 3 3 2 3 4 3 4 2 1 2 1 3 1 1 3 3 3 2 1 3 2 3 4 4 1 2 2 2 2 4 1
## [19657] 4 3 3 2 1 4 3 1 2 2 2 2 3 1 4 3 3 4 1 3 1 4 2 4 4 3 3 4 3 1 4 4 2 3 1 2
## [19693] 2 4 2 2 1 3 2 4 4 4 3 2 1 3 3 2 3 3 1 4 1 4 4 3 3 1 4 1 2 2 2 1 3 1 1 3
## [19729] 3 1 4 4 2 4 4 3 1 4 3 2 4 4 1 3 4 1 1 1 4 3 1 3 2 4 1 3 1 1 4 4 2 3 1 2
## [19765] 1 1 3 4 2 1 1 4 3 2 2 3 3 3 4 1 2 4 1 3 1 3 4 3 4 2 3 4 4 2 4 1 3 1 3 3
## [19801] 2 3 1 4 2 4 1 1 3 2 1 1 1 3 2 1 1 4 3 2 1 4 2 3 1 4 4 3 1 2 3 3 4 1 4 1
## [19837] 4 2 1 1 2 1 3 2 2 1 3 3 3 1 1 1 3 1 3 1 3 3 4 4 1 3 4 3 1 4 3 2 4 1 3 3
## [19873] 1 1 1 3 3 3 3 1 3 3 4 2 2 3 3 3 1 4 4 4 3 2 2 3 1 2 3 2 3 2 2 2 3 3 4 3
## [19909] 4 2 4 2 1 1 1 1 1 1 4 1 3 2 3 1 1 1 3 4 1 2 1 4 4 4 3 1 4 1 4 1 3 4 3 1
## [19945] 1 3 3 3 2 3 2 1 1 1 3 3 3 2 3 1 3 2 1 1 2 4 2 4 4 4 2 1 3 1 3 1 1 1 2 1
## [19981] 3 1 1 3 2 3 3 3 3 3 3 1 3 4 3 3 1 4 4 3 2 1 1 1 4 4 2 4 1 4 2 3 1 3 2 2
## [20017] 1 4 3 3 3 1 1 3 3 1 1 4 3 2 2 1 3 3 4 3 2 2 2 1 4 2 3 2 4 1 2 2 1 4 4 2
## [20053] 3 1 1 3 3 1 2 1 3 1 1 4 3 4 3 1 3 4 2 1 4 3 4 4 4 1 1 3 2 2 4 3 2 3 4 4
## [20089] 4 4 4 4 4 2 1 1 1 3 2 2 3 4 3 2 2 3 2 2 4 4 1 2 2 3 1 2 2 2 3 4 1 1 3 4
## [20125] 3 4 4 3 4 2 1 3 3 3 3 2 3 2 1 3 4 1 3 3 3 3 3 4 3 1 3 3 3 4 1 1 2 2 1 1
## [20161] 2 4 1 2 2 2 2 2 2 3 4 3 1 3 1 1 1 1 3 2 3 3 1 1 1 4 3 3 1 4 3 1 2 1 4 2
## [20197] 3 3 3 2 2 1 4 3 4 1 4 4 3 2 3 4 4 3 1 3 4 3 1 3 3 2 3 2 4 3 1 4 3 3 2 1
## [20233] 2 3 2 2 1 3 2 3 2 4 4 3 4 2 4 3 4 3 2 2 4 1 2 2 4 4 3 3 3 3 1 1 2 1 2 2
## [20269] 1 2 3 4 1 4 2 1 2 3 4 3 1 3 1 2 4 4 3 4 3 3 4 2 4 2 4 1 3 2 1 3 4 1 3 3
## [20305] 4 4 3 2 3 1 1 2 1 3 3 3 1 1 1 2 2 2 4 1 3 4 1 1 3 2 1 3 3 1 1 1 3 2 2 1
## [20341] 4 2 1 1 1 3 2 2 2 1 1 1 4 4 2 3 3 1 2 1 3 2 1 2 3 3 2 3 4 3 2 3 1 3 2 2
## [20377] 2 4 4 2 1 1 1 3 1 2 2 4 1 3 3 3 2 3 2 1 3 1 4 3 4 2 4 2 1 1 2 4 3 4 3 3
## [20413] 2 4 2 4 3 1 3 3 2 2 3 2 2 3 3 3 2 3 1 4 2 3 2 2 1 3 3 1 1 4 3 2 2 3 3 1
## [20449] 4 3 4 2 4 3 3 2 2 1 1 3 3 3 1 2 4 3 1 1 3 4 2 1 1 3 3 1 3 4 3 2 3 1 3 2
## [20485] 3 1 3 3 4 1 3 1 3 3 2 4 2 3 3 2 3 1 1 4 4 4 1 2 2 3 3 4 1 4 2 3 2 3 2 2
## [20521] 3 3 4 1 2 1 3 4 1 4 2 2 2 1 3 1 3 1 3 3 3 3 4 2 3 2 1 3 4 2 3 3 4 3 2 2
## [20557] 4 3 4 3 1 1 3 3 2 3 3 4 4 4 4 2 2 2 3 1 4 3 4 1 2 3 3 1 4 1 1 2 4 4 4 1
## [20593] 1 1 1 3 4 4 3 2 3 4 4 3 2 4 4 3 1 3 3 3 3 4 4 4 3 1 1 1 3 2 4 4 2 2 4 4
## [20629] 1 2 1 3 4 4 3 2 3 4 4 2 2 3 1 3 4 2 1 3 3 3 3 4 3 3 3 1 3 4 1 2 1 1 2 2
## [20665] 1 2 3 1 1 1 1 2 1 2 1 1 4 3 2 3 3 3 2 2 3 1 1 3 3 1 2 1 2 3 3 1 1 2 1 1
## [20701] 3 3 3 3 3 4 4 1 1 1 3 1 1 4 2 3 3 4 2 1 4 1 4 1 1 4 3 2 1 1 2 3 4 3 3 2
## [20737] 2 2 1 1 1 4 1 3 4 3 3 2 3 3 2 2 4 2 1 4 4 3 4 3 3 4 1 1 4 1 3 4 2 2 1 3
## [20773] 1 4 1 3 2 4 4 4 4 3 1 1 4 4 3 4 3 3 4 3 3 3 4 4 4 2 3 2 4 3 2 1 2 2 2 4
## [20809] 4 2 3 2 3 2 4 4 4 2 3 4 2 1 2 4 4 4 4 3 1 1 3 3 1 3 3 3 1 4 3 3 4 3 2 2
## [20845] 1 3 3 2 1 1 4 2 3 3 3 2 2 1 3 1 2 3 1 2 4 1 1 2 2 1 4 4 3 2 1 1 1 3 4 4
## [20881] 3 1 2 3 2 4 4 3 3 3 1 3 2 2 1 3 1 2 4 3 2 2 2 4 1 1 3 3 1 4 2 1 3 4 4 2
## [20917] 1 2 4 2 2 4 3 1 4 2 4 3 3 2 3 1 1 3 1 3 4 3 4 3 3 1 2 2 3 1 3 1 2 4 4 2
## [20953] 4 4 3 4 4 3 1 1 4 3 1 3 4 1 4 3 3 1 3 4 1 2 1 3 2 3 4 4 3 3 1 1 3 3 4 2
## [20989] 2 4 1 3 3 1 2 1 3 1 4 2 4 4 1 4 2 1 2 3 1 1 1 3 2 2 1 4 3 1 3 3 3 4 3 1
## [21025] 3 2 1 3 2 1 1 1 2 1 3 4 3 1 2 3 2 3 4 1 1 2 4 4 3 3 2 2 4 4 4 1 4 1 3 4
## [21061] 1 2 3 2 3 2 4 4 2 4 3 4 3 2 3 3 3 1 4 2 2 2 1 3 4 2 1 4 1 2 4 2 1 3 2 1
## [21097] 4 3 3 4 3 4 2 1 1 2 3 2 3 2 3 1 3 3 3 4 1 1 2 2 4 2 3 4 4 1 4 1 1 4 3 3
## [21133] 4 3 1 3 3 1 3 3 2 3 4 1 4 2 3 2 1 3 3 3 1 3 2 4 3 1 4 1 3 3 2 3 1 3 3 1
## [21169] 3 3 1 1 2 4 1 2 3 3 1 3 4 2 1 4 4 2 1 3 1 3 4 4 3 2 3 4 1 4 3 3 1 3 2 2
## [21205] 3 2 1 3 1 1 2 2 2 2 3 3 4 2 1 1 1 1 1 3 1 2 4 1 3 1 4 4 1 1 3 3 3 4 4 4
## [21241] 2 3 2 2 1 3 1 4 3 3 3 1 3 1 3 1 1 3 2 1 2 4 4 3 4 4 1 1 3 3 2 1 1 1 1 3
## [21277] 3 1 3 1 4 1 3 1 4 3 3 1 3 3 3 3 1 2 1 1 2 1 3 1 3 4 3 3 3 2 3 3 1 2 3 2
## [21313] 1 3 1 2 1 1 1 4 2 3 4 2 1 1 3 2 4 3 1 1 4 3 1 1 1 3 1 1 3 3 2 3 1 4 1 1
## [21349] 3 3 3 4 3 2 4 3 4 3 2 2 4 2 2 2 3 2 3 2 2 2 4 4 4 2 3 2 4 2 3 4 2 4 1 3
## [21385] 1 3 3 3 4 4 4 1 2 3 2 3 2 1 3 3 2 1 4 1 4 2 1 1 3 4 2 3 2 3 3 1 2 3 3 4
## [21421] 1 1 2 4 4 4 4 4 3 2 2 4 4 4 1 1 4 4 2 2 2 4 2 1 3 3 1 4 2 1 4 4 2 3 2 3
## [21457] 4 2 3 2 4 1 4 4 4 4 4 3 4 1 1 2 1 2 1 4 1 1 1 1 1 3 3 1 3 2 2 4 4 1 3 4
## [21493] 2 4 4 3 1 1 3 2 3 4 1 1 3 1 2 3 1 3 2 2 3 4 3 4 4 3 1 2 4 3 3 4 3 1 1 1
## [21529] 1 1 4 3 3 3 3 1 4 1 3 3 2 4 2 3 1 1 1 1 1 3 2 1 3 2 4 2 3 1 2 1 1 3 1 3
## [21565] 3 4 3 1 1 2 2 4 4 4 4 3 3 2 2 3 1 3 1 3 4 3 2 4 3 3 3 4 1 3 3 1 3 1 1 2
## [21601] 3 1 4 3 1 1 1 2 4 4 4 4 2 3 2 4 3 3 4 4 2 1 1 2 1 2 4 1 2 2 1 3 2 4 2 1
## [21637] 3 2 3 4 3 3 3 3 4 2 3 4 3 4 1 1 2 3 1 3 2 1 3 3 4 3 4 1 4 2 3 2 2 2 3 3
## [21673] 4 3 3 2 1 1 1 1 4 2 4 3 4 3 2 3 2 3 1 2 3 1 3 3 1 4 1 1 2 3 3 2 1 2 2 2
## [21709] 4 3 3 1 1 1 4 4 1 1 2 4 1 2 2 3 3 3 2 1 2 1 1 3 3 2 2 1 3 1 1 1 1 2 1 3
## [21745] 2 1 4 2 2 3 3 2 4 4 4 4 3 4 4 3 3 3 2 4 4 1 1 2 1 3 1 3 1 1 3 4 3 2 3 2
## [21781] 2 1 1 1 2 2 4 2 2 2 3 3 1 1 3 3 1 3 4 4 4 2 1 4 2 1 3 2 4 2 2 3 2 2 4 4
## [21817] 1 1 4 1 1 3 3 4 3 2 1 2 2 3 3 4 1 1 1 1 4 1 1 2 1 4 1 1 3 3 2 2 3 3 4 3
## [21853] 2 1 1 2 2 4 4 2 2 3 3 1 1 3 4 4 3 4 4 4 2 2 2 2 2 4 4 4 3 3 2 1 3 1 4 3
## [21889] 3 3 1 2 4 4 4 3 1 3 3 3 3 3 1 4 1 4 4 2 1 3 2 3 1 3 4 4 3 1 2 3 3 3 1 2
## [21925] 3 1 2 1 4 1 4 2 1 3 4 1 4 3 2 3 3 2 2 1 3 2 3 3 2 2 3 2 4 3 1 4 3 2 3 3
## [21961] 3 4 4 1 3 4 2 3 3 2 1 1 4 1 3 3 3 1 1 4 3 2 2 2 4 1 2 4 4 4 1 3 1 1 3 4
## [21997] 1 3 1 2 2 2 4 3 2 3 1 3 2 1 3 1 4 1 1 2 3 1 3 4 3 3 3 3 3 2 3 1 3 3 3 4
## [22033] 3 3 2 3 2 3 3 2 2 1 4 1 3 3 3 4 4 1 3 3 2 2 1 1 2 2 4 2 4 2 1 1 2 3 4 1
## [22069] 3 4 2 2 4 1 2 2 4 4 1 1 4 2 3 1 1 1 1 1 3 4 2 1 1 4 2 1 2 3 2 3 1 1 1 2
## [22105] 3 4 4 4 1 1 1 3 1 1 4 1 4 2 4 4 3 4 3 2 4 3 2 4 2 1 1 1 2 1 1 3 3 3 3 1
## [22141] 1 4 4 3 2 1 4 1 2 3 4 2 4 2 1 2 3 1 2 3 1 3 3 1 4 2 4 1 2 1 3 3 2 4 1 4
## [22177] 4 4 1 1 1 3 1 2 3 1 2 3 3 4 4 3 2 2 3 4 1 3 1 3 2 1 4 2 2 4 3 3 1 4 1 2
## [22213] 4 1 3 3 3 1 2 1 4 3 4 3 4 4 4 3 3 2 2 1 2 4 3 1 1 2 2 1 3 2 2 1 3 2 2 2
## [22249] 2 3 3 4 4 4 4 1 2 3 3 4 4 1 3 3 1 4 4 2 2 2 2 2 1 3 1 3 2 3 1 4 2 2 2 4
## [22285] 3 3 3 3 4 3 3 3 3 3 3 1 1 1 4 4 2 3 2 1 2 3 1 2 1 3 3 2 4 3 2 1 2 3 4 3
## [22321] 1 3 1 4 4 1 3 2 3 4 3 4 1 3 1 2 1 4 4 1 1 4 4 3 3 3 4 4 1 1 2 2 3 3 2 4
## [22357] 4 2 4 2 1 2 4 4 2 3 3 3 4 1 4 4 3 4 3 1 4 3 4 4 2 3 3 2 3 1 4 1 4 3 1 2
## [22393] 3 1 1 3 2 3 1 1 3 4 3 4 4 4 1 2 4 1 3 4 1 3 4 2 2 3 1 3 1 1 2 2 1 1 1 2
## [22429] 1 4 3 2 1 3 3 4 3 2 3 4 1 2 1 1 4 1 2 3 3 4 3 3 4 3 2 1 3 3 3 1 2 1 3 4
## [22465] 4 1 3 3 2 3 1 3 3 2 4 3 4 3 4 4 3 4 3 4 1 1 3 4 3 1 3 3 4 4 1 4 3 1 2 3
## [22501] 4 4 3 2 3 3 3 3 4 3 2 4 1 3 4 3 3 3 1 2 4 2 2 4 3 3 4 1 1 3 1 3 2 1 2 4
## [22537] 3 1 1 3 3 4 1 1 3 1 4 4 4 1 2 4 2 1 3 2 3 1 3 4 4 4 4 3 4 4 2 4 1 1 4 4
## [22573] 1 1 4 3 1 4 2 1 2 3 1 3 4 3 2 1 3 3 2 2 4 2 3 3 4 4 1 1 1 3 3 4 3 4 4 4
## [22609] 3 4 3 1 1 2 3 4 1 2 3 2 3 4 1 2 4 4 3 3 3 3 4 1 4 3 2 3 1 3 1 3 1 3 1 2
## [22645] 1 4 2 3 2 4 3 1 3 2 4 4 3 3 4 2 1 3 1 4 2 3 4 4 2 3 2 3 3 3 4 4 2 2 3 1
## [22681] 3 3 3 2 2 1 1 2 2 1 4 1 2 3 4 2 1 4 4 2 2 4 1 3 3 2 4 1 4 1 4 1 4 2 1 3
## [22717] 3 2 3 3 2 4 3 2 4 1 3 2 1 3 3 1 1 2 4 4 2 4 2 3 2 1 2 3 3 2 1 2 3 4 3 3
## [22753] 2 1 2 4 1 1 3 2 1 1 1 1 3 3 2 1 1 3 4 1 4 2 2 4 2 3 2 3 2 4 3 2 1 1 4 2
## [22789] 4 4 4 4 3 3 1 2 1 3 1 2 4 2 1 4 4 3 1 4 3 3 2 4 3 3 3 2 4 2 3 3 3 1 2 2
## [22825] 3 3 2 4 3 4 4 2 2 1 1 3 2 3 4 2 1 3 2 3 1 2 3 4 3 2 3 1 3 4 2 3 2 2 3 4
## [22861] 2 4 2 1 4 2 4 4 1 2 1 4 1 4 2 3 1 2 4 3 1 4 2 3 3 3 3 1 2 1 2 2 1 2 3 1
## [22897] 2 1 3 3 4 3 3 1 4 3 3 2 1 3 3 4 3 3 1 1 1 4 2 3 2 3 1 2 2 1 2 4 1 1 1 2
## [22933] 4 3 3 2 1 4 2 4 1 3 1 4 4 2 1 4 1 1 4 4 3 4 3 3 4 2 2 1 4 1 2 2 2 3 3 1
## [22969] 4 1 4 1 3 3 3 3 2 2 1 3 4 2 2 1 3 3 4 4 2 3 3 4 3 3 4 4 2 4 2 4 1 2 3 4
## [23005] 3 2 4 2 2 3 2 2 3 1 3 2 3 2 4 3 1 2 3 2 1 3 2 2 3 4 2 1 2 3 2 2 2 4 4 3
## [23041] 2 2 3 2 3 2 2 4 2 2 3 2 1 3 1 3 1 1 3 3 2 1 4 4 1 4 3 2 1 3 1 4 2 4 2 4
## [23077] 1 3 4 4 3 4 2 3 3 4 2 3 3 3 3 2 1 1 4 2 1 3 4 2 4 4 3 3 1 4 1 4 4 4 1 1
## [23113] 1 3 2 1 2 2 4 4 1 2 3 2 2 2 4 3 3 1 4 3 1 3 4 3 2 4 4 3 3 1 1 2 3 3 2 4
## [23149] 3 3 4 1 4 4 1 4 1 4 4 1 4 1 2 2 3 4 2 4 1 3 1 3 1 1 3 1 4 4 4 2 3 4 1 2
## [23185] 1 3 4 1 1 1 1 1 3 3 4 1 4 4 2 3 4 2 3 3 1 4 2 3 3 4 3 1 2 4 1 3 2 1 1 1
## [23221] 3 3 4 3 3 3 4 4 4 1 2 1 2 1 3 1 4 4 1 2 3 1 4 3 4 1 3 4 2 3 2 1 1 4 4 4
## [23257] 3 4 1 2 4 3 2 4 3 4 3 1 1 3 4 2 4 2 1 2 3 4 2 4 4 2 1 1 1 2 1 1 3 1 1 4
## [23293] 4 3 2 4 4 2 4 1 3 2 4 3 1 1 3 1 3 3 3 1 3 4 3 2 4 3 3 4 3 2 3 3 4 2 4 4
## [23329] 4 3 1 2 3 3 3 1 2 4 4 1 4 3 3 4 1 1 2 4 4 3 4 3 4 3 3 3 2 3 1 4 1 4 3 3
## [23365] 2 4 3 1 1 1 2 4 4 3 3 3 1 3 4 1 1 2 3 1 1 3 1 2 4 3 3 2 2 4 3 3 2 1 4 4
## [23401] 4 3 1 3 3 3 1 4 4 4 3 1 4 3 2 3 2 3 4 2 4 4 3 3 3 2 4 3 1 3 3 2 3 2 4 2
## [23437] 3 3 1 3 3 4 3 1 3 1 1 2 3 1 1 2 2 2 4 3 3 3 4 3 3 2 3 2 1 4 3 3 4 3 3 3
## [23473] 2 1 1 2 1 3 1 3 1 3 2 4 1 3 1 4 2 1 1 2 3 3 4 1 3 2 1 3 2 3 1 4 3 3 3 2
## [23509] 2 4 3 4 1 1 1 4 4 3 4 1 4 1 3 4 2 1 4 2 4 4 1 2 1 1 4 3 3 1 1 2 3 3 1 3
## [23545] 3 1 3 4 3 3 1 3 4 4 2 1 3 4 3 4 4 2 3 1 1 2 1 1 1 2 2 2 1 1 3 2 2 4 4 2
## [23581] 3 3 1 3 1 4 3 1 1 2 1 3 2 2 1 2 4 1 3 3 2 2 1 2 2 1 2 3 4 3 2 1 2 4 2 3
## [23617] 4 2 1 3 2 2 4 3 2 2 3 1 2 1 1 3 2 4 1 3 1 2 3 3 3 1 4 4 3 1 4 1 4 2 4 3
## [23653] 1 3 2 4 4 1 4 2 4 3 4 2 1 2 4 2 1 4 2 3 2 1 2 3 3 1 1 4 2 3 1 3 3 3 1 3
## [23689] 1 1 3 3 4 4 3 1 1 1 2 4 3 4 2 2 2 3 3 1 1 3 3 1 4 1 3 3 2 2 2 3 3 4 1 1
## [23725] 2 3 2 2 3 2 4 1 2 3 3 3 1 3 2 3 3 4 4 3 3 1 2 3 1 4 3 1 3 4 2 2 3 4 1 4
## [23761] 4 3 1 3 3 4 3 2 1 1 3 3 3 1 3 2 2 1 2 3 1 3 2 1 4 3 4 3 1 2 2 3 3 3 1 1
## [23797] 1 2 1 1 3 1 1 2 4 3 1 2 1 1 2 2 1 1 3 3 3 1 1 3 3 2 3 1 2 1 1 2 4 3 1 4
## [23833] 3 2 1 3 2 3 1 1 3 4 3 3 4 3 3 4 1 1 1 1 3 1 3 1 4 4 3 3 1 2 1 2 3 1 4 4
## [23869] 3 3 4 1 1 1 1 3 4 4 1 1 4 2 1 3 4 1 3 4 2 4 1 1 4 4 1 2 3 4 2 4 2 2 3 3
## [23905] 3 3 2 4 1 3 4 4 1 3 1 1 4 3 2 1 3 1 4 4 2 2 4 3 1 4 3 4 1 3 2 1 3 1 3 3
## [23941] 4 3 2 2 2 1 2 2 2 2 2 3 2 1 2 2 3 1 1 1 4 3 2 4 4 2 3 3 2 1 4 1 1 3 2 1
## [23977] 3 2 3 4 4 3 1 2 1 4 1 3 3 4 1 2 2 1 3 3 4 3 4 3 4 3 3 4 2 2 2 3 3 1 3 1
## [24013] 2 1 3 4 4 2 4 1 3 1 2 4 4 2 1 4 1 3 3 4 3 1 2 3 3 1 2 2 2 2 4 3 2 2 1 4
## [24049] 3 2 3 1 1 3 4 3 3 3 2 2 1 3 3 1 4 2 1 2 4 2 3 2 2 3 4 2 4 2 1 3 1 2 2 1
## [24085] 4 4 4 4 4 3 1 3 3 1 4 4 1 3 1 1 3 2 3 1 1 2 2 4 4 1 3 3 4 4 1 4 1 2 1 3
## [24121] 1 1 4 1 4 4 4 1 4 4 3 3 1 3 3 1 2 3 4 4 2 4 3 2 3 1 4 1 3 1 3 3 3 3 3 1
## [24157] 1 4 1 1 3 2 3 4 1 2 4 4 2 3 2 3 2 2 2 2 3 3 4 1 1 1 2 1 3 1 1 1 3 1 2 3
## [24193] 3 1 4 3 1 4 2 2 4 2 1 1 4 2 2 4 3 3 3 3 2 4 4 3 4 2 4 1 2 3 2 1 3 3 2 2
## [24229] 1 3 3 4 3 1 2 3 4 3 1 3 4 3 3 2 4 4 1 2 1 2 1 2 4 1 1 3 4 3 1 4 2 1 2 2
## [24265] 3 2 3 3 1 1 4 1 2 4 2 3 2 1 3 2 2 3 3 4 4 2 3 4 4 3 4 4 4 2 3 3 1 3 3 4
## [24301] 4 3 3 2 1 3 4 3 4 4 1 3 3 1 3 3 4 2 3 3 1 2 2 3 2 4 3 3 1 4 2 1 4 3 2 2
## [24337] 3 4 2 2 1 1 4 3 1 4 2 1 1 2 2 4 4 3 1 4 3 4 4 3 1 1 1 1 3 3 2 1 3 4 4 1
## [24373] 3 4 1 1 3 1 2 2 2 3 2 3 3 3 4 3 1 2 3 2 3 4 2 2 3 1 1 4 2 2 4 3 1 4 3 2
## [24409] 4 4 4 4 1 2 3 2 3 3 1 4 2 4 3 2 3 2 3 4 4 1 3 1 3 1 2 3 1 4 1 1 2 4 1 3
## [24445] 3 4 4 1 2 2 2 3 2 4 1 2 3 1 1 2 3 3 4 2 4 1 4 4 1 3 1 1 3 1 3 3 1 1 4 3
## [24481] 1 3 3 1 2 1 2 4 4 4 3 3 3 2 3 4 3 3 2 3 3 2 4 2 4 4 4 1 3 3 4 3 3 3 4 1
## [24517] 3 3 1 4 1 3 3 2 3 3 4 1 3 4 1 2 3 3 4 4 3 1 1 1 2 4 3 4 3 2 4 2 4 1 3 4
## [24553] 1 3 1 3 1 2 4 3 4 3 2 3 2 1 3 3 3 2 3 4 3 3 1 2 3 2 1 3 2 2 1 2 1 3 1 4
## [24589] 3 1 4 3 4 1 3 1 3 3 1 1 4 1 1 4 3 3 1 4 3 3 4 3 4 1 2 4 3 1 1 2 3 2 1 2
## [24625] 3 1 3 2 2 2 2 1 3 3 4 3 3 1 4 2 1 3 1 3 2 1 4 1 3 3 4 4 4 4 2 3 3 1 4 4
## [24661] 1 4 2 3 2 2 2 2 2 3 1 3 3 3 1 4 1 1 4 3 3 4 3 4 3 1 2 4 1 2 3 1 1 3 2 1
## [24697] 1 2 1 3 1 2 4 4 2 1 1 1 4 3 2 3 4 3 2 3 4 2 1 4 1 3 4 3 4 1 1 3 2 3 3 2
## [24733] 2 1 2 4 3 3 4 1 1 4 2 3 4 1 4 4 2 4 3 1 3 2 2 1 3 2 1 4 1 3 3 3 3 3 2 1
## [24769] 1 4 4 1 1 2 3 1 1 1 2 1 3 2 4 3 1 3 4 4 2 1 2 3 2 2 1 2 4 2 2 1 2 2 3 3
## [24805] 3 2 1 3 2 3 4 4 1 3 4 4 4 4 2 3 2 4 1 4 2 4 4 1 2 2 3 2 2 3 2 3 4 2 2 1
## [24841] 4 4 2 1 4 4 3 3 1 1 4 1 4 3 1 4 1 3 4 3 2 3 3 1 3 1 1 3 4 1 1 3 4 3 4 4
## [24877] 3 4 4 1 3 3 2 1 3 4 2 2 2 2 1 3 1 3 3 4 4 1 3 4 3 3 2 1 1 4 2 1 1 3 2 4
## [24913] 4 4 1 4 1 3 1 4 1 3 3 3 3 3 2 3 2 2 2 3 2 1 3 2 1 1 1 4 1 2 2 2 4 4 3 1
## [24949] 2 4 4 4 3 1 3 2 4 2 2 2 2 2 1 1 3 1 1 3 4 4 3 4 2 4 4 2 2 1 3 3 1 4 1 3
## [24985] 2 4 2 1 4 4 3 1 2 1 2 1 1 2 2 3 1 2 3 1 1 3 2 4 1 1 3 2 2 1 4 3 1 2 1 3
## [25021] 4 2 3 1 4 2 4 4 4 1 4 4 3 3 1 2 4 4 1 2 4 3 3 1 4 3 3 2 1 3 2 1 3 2 1 3
## [25057] 1 3 2 2 3 4 3 2 4 3 3 2 3 1 2 1 1 3 1 3 4 1 1 3 4 3 1 2 3 4 2 1 3 3 2 3
## [25093] 4 2 2 2 2 3 1 1 3 2 1 2 1 3 4 2 1 3 1 2 4 1 3 4 2 3 4 4 4 3 2 2 3 3 1 4
## [25129] 1 3 4 4 4 4 2 1 3 3 1 2 1 2 2 4 3 3 1 2 4 4 3 3 3 4 3 2 2 2 1 3 3 1 2 3
## [25165] 3 1 1 2 2 2 1 2 2 1 3 4 2 3 1 4 1 2 1 1 1 1 1 4 4 2 4 2 2 3 3 3 4 3 2 3
## [25201] 3 4 3 4 3 4 1 3 2 3 1 3 3 3 3 3 4 1 1 2 4 3 1 1 3 1 4 4 1 1 2 3 3 2 2 2
## [25237] 1 3 1 1 3 3 1 2 2 1 4 1 2 3 1 2 1 4 1 3 3 4 3 3 3 3 3 3 3 2 2 4 3 3 3 4
## [25273] 2 3 4 3 1 2 3 4 1 1 3 1 2 2 4 2 1 2 3 4 3 4 1 1 2 1 3 4 2 3 3 1 3 3 4 3
## [25309] 1 1 4 3 2 3 4 2 1 3 1 1 4 4 3 2 2 4 1 2 1 4 4 4 2 2 4 4 1 4 3 1 4 4 1 3
## [25345] 3 2 4 2 4 2 4 3 3 4 3 3 1 3 2 4 3 2 1 3 3 3 2 1 3 1 2 1 3 1 2 4 1 2 1 4
## [25381] 3 4 1 3 3 2 1 2 1 1 3 4 2 4 4 4 4 2 3 2 4 3 3 2 3 2 3 1 3 2 1 3 4 3 2 4
## [25417] 1 3 2 4 4 3 4 1 2 1 3 1 3 2 3 3 1 4 3 3 2 4 3 2 2 4 2 3 4 3 3 4 1 1 3 1
## [25453] 1 2 3 3 3 1 3 3 1 3 1 4 3 4 1 1 2 1 1 2 3 3 1 2 3 1 4 1 1 4 1 3 4 3 2 3
## [25489] 4 3 3 1 4 4 4 2 3 2 3 2 3 1 1 3 2 2 2 3 4 2 2 2 2 3 3 3 3 2 4 3 1 4 3 1
## [25525] 1 3 2 3 4 3 3 1 4 3 2 1 2 1 4 3 1 3 3 1 3 1 2 2 4 2 2 3 3 2 1 1 1 3 2 1
## [25561] 4 4 3 1 3 2 2 1 4 1 3 1 4 2 3 4 1 1 2 2 1 1 4 2 3 3 4 3 3 2 4 3 1 1 3 1
## [25597] 2 4 2 4 3 3 1 1 3 4 3 2 1 4 1 4 1 3 4 3 1 1 1 3 4 2 4 1 1 2 3 4 3 4 3 3
## [25633] 1 4 4 1 3 1 3 4 3 3 3 2 1 1 4 4 2 4 2 1 2 1 4 3 3 1 4 3 4 2 4 3 3 3 1 1
## [25669] 3 3 1 4 1 4 4 3 2 2 4 3 4 3 2 1 1 3 1 1 4 3 1 1 2 3 3 2 4 1 4 2 4 3 1 4
## [25705] 3 2 3 4 1 2 4 2 2 1 4 4 2 4 4 4 3 2 3 4 3 2 1 2 1 2 3 2 1 1 2 3 4 3 4 3
## [25741] 4 3 4 2 3 4 1 4 3 2 2 2 1 2 1 3 4 2 2 2 2 3 3 3 2 1 3 2 4 3 1 3 3 3 4 1
## [25777] 1 3 1 1 1 3 2 2 4 4 3 3 3 3 4 4 3 2 1 3 1 1 1 3 1 3 3 1 2 3 3 1 1 1 1 1
## [25813] 1 3 2 1 2 1 3 3 4 2 1 4 4 3 2 2 2 3 4 2 2 1 2 1 1 3 4 2 3 4 1 1 3 1 4 3
## [25849] 2 1 1 4 3 4 3 3 3 3 1 3 1 1 4 1 3 3 4 3 4 2 3 4 4 1 1 3 4 1 3 2 4 2 4 1
## [25885] 3 4 2 3 3 2 1
final_predicted = factor(final_predicted, levels=c(1,2,3,4,5))
actual_labels = factor(airline_test_set_filtered$Inflight.service, levels=c(1,2,3,4,5))
confusionMatrix(final_predicted, factor(actual_labels))
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    1    2    3    4    5
##          1 1197 2012 1584 1350  421
##          2  192  305 2424 1647 1122
##          3  214  331  803 5255 1211
##          4  172  190  194 1100 4167
##          5    0    0    0    0    0
## 
## Overall Statistics
##                                           
##                Accuracy : 0.1315          
##                  95% CI : (0.1274, 0.1357)
##     No Information Rate : 0.3612          
##     P-Value [Acc > NIR] : 1               
##                                           
##                   Kappa : -0.0605         
##                                           
##  Mcnemar's Test P-Value : <2e-16          
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity           0.67437  0.10747  0.16044  0.11762   0.0000
## Specificity           0.77745  0.76641  0.66432  0.71443   1.0000
## Pos Pred Value        0.18236  0.05360  0.10276  0.18891      NaN
## Neg Pred Value        0.97009  0.87461  0.76755  0.58880   0.7327
## Prevalence            0.06856  0.10961  0.19331  0.36121   0.2673
## Detection Rate        0.04623  0.01178  0.03101  0.04249   0.0000
## Detection Prevalence  0.25352  0.21977  0.30180  0.22490   0.0000
## Balanced Accuracy     0.72591  0.43694  0.41238  0.41603   0.5000

regularization term을 포함해서 모델을 fitting 후에, 정규화 항이 없는 모델과 비교를 한 결과, 오히려 test data set에서의 민감도에 대한 성능이 더 낮아졌다. 제일 높은 민감도값을 기준으로 0.7380 -> 0.5619(Class 4) 감소했다.

natural spline model

table(airline_train_set$Inflight.service)
## 
##     0     1     2     3     4     5 
##     3  7063 11414 20227 37846 27041
table(airline_test_set$Inflight.service)
## 
##    0    1    2    3    4    5 
##    2 1775 2838 5005 9352 6921

확인결과, 0에 대한 class의 수가 매우 부족한 것을 알 수 있다. 0에 대한 클래스 데이터를 제거하기로 한다.

airline_test_set_filtered = airline_test_set[airline_test_set$Inflight.service != 0, ]
airline_train_set_filtered = airline_train_set[airline_train_set$Inflight.service != 0, ]

table(airline_test_set_filtered$Inflight.service)
## 
##    1    2    3    4    5 
## 1775 2838 5005 9352 6921
table(airline_train_set_filtered$Inflight.service)
## 
##     1     2     3     4     5 
##  7063 11414 20227 37846 27041

종속변수의 클래스가 0인 경우가 제거된 것을 확인할 수 있다.

library(splines)

max_degree = 10
cv_errors_sp = rep(NA, max_degree)

trainX = model.matrix(~ ns(On.board.service, df = 4), data = airline_train_set_filtered)[, -1]
trainY = airline_train_set_filtered$Inflight.service
testX = model.matrix(~ ns(On.board.service, df = 4), data = airline_test_set_filtered)[, -1]
fit = cv.glmnet(trainX, trainY, family= "multinomial")
best_lambda = fit$lambda.min
print(best_lambda)
## [1] 0.0006734283

cv를 통해서 얻은 최선의 lambda 값으로 model을 fitting 한다.

final_sp_model = glmnet(trainX, trainY, fmaily = "multinomial", lambda = best_lambda)
prediction_sp = predict(final_sp_model, newx = testX, s = best_lambda, type = "class")
prediction_sp = round(prediction_sp)
actual = airline_test_set_filtered$Inflight.service

prediction_sp = factor(prediction_sp, levels = c(1,2,3,4,5))
actual = factor(actual, levels = c(1,2,3,4,5))
#confusion matrix
confusionMatrix(prediction_sp, factor(actual))
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    1    2    3    4    5
##          1    0    0    0    0    0
##          2    0    0    0    0    0
##          3 1197 2012 1584 1350  421
##          4  406  636 3227 6902 2333
##          5  172  190  194 1100 4167
## 
## Overall Statistics
##                                           
##                Accuracy : 0.4887          
##                  95% CI : (0.4826, 0.4948)
##     No Information Rate : 0.3612          
##     P-Value [Acc > NIR] : < 2.2e-16       
##                                           
##                   Kappa : 0.2721          
##                                           
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4 Class: 5
## Sensitivity           0.00000   0.0000  0.31648   0.7380   0.6021
## Specificity           1.00000   1.0000  0.76156   0.6008   0.9127
## Pos Pred Value            NaN      NaN  0.24132   0.5111   0.7156
## Neg Pred Value        0.93144   0.8904  0.82299   0.8022   0.8628
## Prevalence            0.06856   0.1096  0.19331   0.3612   0.2673
## Detection Rate        0.00000   0.0000  0.06118   0.2666   0.1609
## Detection Prevalence  0.00000   0.0000  0.25352   0.5216   0.2249
## Balanced Accuracy     0.50000   0.5000  0.53902   0.6694   0.7574

Conclusion

Inflight.service 예측에 대한 각 모델의 accuracy를 비교한다.


1. Linear regression model : 0.5917
2. polynomial regression model(order : 4) : 0.4887
3. Polynomial regression model with regularization(ridge) : 0.5352
4. natural spline model : 0.4887
결과적으로는 linear regression model의 정확도가 가장 높았다. regularization term을 사용한 다항모델의 경우, 그렇지 않은 모델보다 높은 정확도를 보였지만, 해당 모델 역시 선형모델의 성능을 넘어서지는 못했다.

Tree model : satisfaction 예측

우선, 모든 변수를 이용해서 tree 모델을 생성한다.

tree_model_all = rpart(satisfaction ~ . -Flight.Distance -Departure.Delay.in.Minutes -Arrival.Delay.in.Minutes, data = airline_train_set)
summary(tree_model_all)
## Call:
## rpart(formula = satisfaction ~ . - Flight.Distance - Departure.Delay.in.Minutes - 
##     Arrival.Delay.in.Minutes, data = airline_train_set)
##   n= 103594 
## 
##           CP nsplit rel error    xerror         xstd
## 1 0.33564556      0 1.0000000 1.0000242 0.0008352578
## 2 0.12871693      1 0.6643544 0.6643802 0.0029027750
## 3 0.05427203      2 0.5356375 0.5356761 0.0031438054
## 4 0.02957519      4 0.4270934 0.4271468 0.0029321065
## 5 0.02820567      5 0.3975183 0.3975665 0.0029275042
## 6 0.01904935      6 0.3693126 0.3693661 0.0027334211
## 7 0.01490694      7 0.3502632 0.3503195 0.0027208857
## 8 0.01055337     10 0.3055424 0.3056090 0.0024667627
## 9 0.01000000     11 0.2949890 0.2959254 0.0024837119
## 
## Variable importance
##        Online.boarding  Inflight.wifi.service           Seat.comfort 
##                     24                     19                     11 
## Inflight.entertainment Ease.of.Online.booking                  Class 
##                     10                     10                      9 
##         Type.of.Travel            Cleanliness          Customer.Type 
##                      9                      1                      1 
##       On.board.service                    Age        Checkin.service 
##                      1                      1                      1 
##         Food.and.drink       Inflight.service       Leg.room.service 
##                      1                      1                      1 
## 
## Node number 1: 103594 observations,    complexity param=0.3356456
##   mean=0.4333938, MSE=0.2455636 
##   left son=2 (52271 obs) right son=3 (51323 obs)
##   Primary splits:
##       Online.boarding        < 3.5      to the left,  improve=0.3356456, (0 missing)
##       Class                  splits as  RLL,          improve=0.2539593, (0 missing)
##       Type.of.Travel         splits as  RL,           improve=0.2015964, (0 missing)
##       Inflight.entertainment < 3.5      to the left,  improve=0.1726111, (0 missing)
##       Inflight.wifi.service  < 3.5      to the left,  improve=0.1676912, (0 missing)
##   Surrogate splits:
##       Seat.comfort           < 3.5      to the left,  agree=0.730, adj=0.455, (0 split)
##       Inflight.wifi.service  < 3.5      to the left,  agree=0.714, adj=0.424, (0 split)
##       Ease.of.Online.booking < 3.5      to the left,  agree=0.676, adj=0.346, (0 split)
##       Class                  splits as  RLL,          agree=0.670, adj=0.334, (0 split)
##       Inflight.entertainment < 3.5      to the left,  agree=0.659, adj=0.311, (0 split)
## 
## Node number 2: 52271 observations,    complexity param=0.05427203
##   mean=0.1489162, MSE=0.1267402 
##   left son=4 (50507 obs) right son=5 (1764 obs)
##   Primary splits:
##       Inflight.wifi.service  < 0.5      to the right, improve=0.19748610, (0 missing)
##       Class                  splits as  RLL,          improve=0.09211006, (0 missing)
##       Ease.of.Online.booking < 0.5      to the right, improve=0.08452210, (0 missing)
##       Online.boarding        < 0.5      to the right, improve=0.06366470, (0 missing)
##       Type.of.Travel         splits as  RL,           improve=0.06305692, (0 missing)
##   Surrogate splits:
##       Ease.of.Online.booking < 0.5      to the right, agree=0.971, adj=0.151, (0 split)
##       Online.boarding        < 0.5      to the right, agree=0.968, adj=0.041, (0 split)
##       Cleanliness            < 0.5      to the right, agree=0.966, adj=0.003, (0 split)
##       Inflight.entertainment < 0.5      to the right, agree=0.966, adj=0.002, (0 split)
## 
## Node number 3: 51323 observations,    complexity param=0.1287169
##   mean=0.7231261, MSE=0.2002148 
##   left son=6 (10451 obs) right son=7 (40872 obs)
##   Primary splits:
##       Type.of.Travel         splits as  RL,           improve=0.3186590, (0 missing)
##       Class                  splits as  RLL,          improve=0.2299112, (0 missing)
##       Inflight.entertainment < 3.5      to the left,  improve=0.1482971, (0 missing)
##       Leg.room.service       < 3.5      to the left,  improve=0.1452057, (0 missing)
##       On.board.service       < 3.5      to the left,  improve=0.0990183, (0 missing)
##   Surrogate splits:
##       Class            splits as  RLR,          agree=0.822, adj=0.127, (0 split)
##       Age              < 60.5     to the right, agree=0.814, adj=0.085, (0 split)
##       Leg.room.service < 0.5      to the left,  agree=0.801, adj=0.022, (0 split)
## 
## Node number 4: 50507 observations,    complexity param=0.05427203
##   mean=0.1193498, MSE=0.1051054 
##   left son=8 (45789 obs) right son=9 (4718 obs)
##   Primary splits:
##       Inflight.wifi.service  < 3.5      to the left,  improve=0.27369580, (0 missing)
##       Class                  splits as  RLL,          improve=0.10150640, (0 missing)
##       Ease.of.Online.booking < 3.5      to the left,  improve=0.07538054, (0 missing)
##       Type.of.Travel         splits as  RL,           improve=0.06892632, (0 missing)
##       Leg.room.service       < 3.5      to the left,  improve=0.06340985, (0 missing)
## 
## Node number 5: 1764 observations
##   mean=0.9954649, MSE=0.00451458 
## 
## Node number 6: 10451 observations,    complexity param=0.02957519
##   mean=0.223615, MSE=0.1736113 
##   left son=12 (9336 obs) right son=13 (1115 obs)
##   Primary splits:
##       Inflight.wifi.service        < 4.5      to the left,  improve=0.41465830, (0 missing)
##       Ease.of.Online.booking       < 4.5      to the left,  improve=0.25583170, (0 missing)
##       Age                          < 41.5     to the right, improve=0.04362807, (0 missing)
##       Arrival.Delay.in.Minutes_log < 1.868835 to the right, improve=0.03946246, (0 missing)
##       Leg.room.service             < 3.5      to the left,  improve=0.03694485, (0 missing)
##   Surrogate splits:
##       Ease.of.Online.booking < 4.5      to the left,  agree=0.949, adj=0.518, (0 split)
## 
## Node number 7: 40872 observations,    complexity param=0.02820567
##   mean=0.8508514, MSE=0.1269033 
##   left son=14 (9989 obs) right son=15 (30883 obs)
##   Primary splits:
##       Inflight.entertainment < 3.5      to the left,  improve=0.1383364, (0 missing)
##       On.board.service       < 3.5      to the left,  improve=0.1181314, (0 missing)
##       Inflight.service       < 3.5      to the left,  improve=0.1163792, (0 missing)
##       Baggage.handling       < 3.5      to the left,  improve=0.1120047, (0 missing)
##       Seat.comfort           < 3.5      to the left,  improve=0.1105292, (0 missing)
##   Surrogate splits:
##       Inflight.service < 3.5      to the left,  agree=0.821, adj=0.267, (0 split)
##       Seat.comfort     < 3.5      to the left,  agree=0.820, adj=0.265, (0 split)
##       Cleanliness      < 2.5      to the left,  agree=0.813, adj=0.233, (0 split)
##       Baggage.handling < 3.5      to the left,  agree=0.810, adj=0.223, (0 split)
##       On.board.service < 3.5      to the left,  agree=0.797, adj=0.170, (0 split)
## 
## Node number 8: 45789 observations,    complexity param=0.01490694
##   mean=0.06490642, MSE=0.06069358 
##   left son=16 (32928 obs) right son=17 (12861 obs)
##   Primary splits:
##       Class                             splits as  RLL,          improve=0.09851169, (0 missing)
##       Type.of.Travel                    splits as  RL,           improve=0.05430057, (0 missing)
##       Flight.Distance_log               < 7.376821 to the left,  improve=0.05282616, (0 missing)
##       Leg.room.service                  < 3.5      to the left,  improve=0.03067756, (0 missing)
##       Departure.Arrival.time.convenient < 3.5      to the right, improve=0.03004393, (0 missing)
##   Surrogate splits:
##       Flight.Distance_log          < 7.470509 to the left,  agree=0.778, adj=0.208, (0 split)
##       Ease.of.Online.booking       < 3.5      to the left,  agree=0.723, adj=0.012, (0 split)
##       Age                          < 78.5     to the left,  agree=0.719, adj=0.001, (0 split)
##       Arrival.Delay.in.Minutes_log < 6.547471 to the left,  agree=0.719, adj=0.000, (0 split)
##       Inflight.entertainment       < 0.5      to the right, agree=0.719, adj=0.000, (0 split)
## 
## Node number 9: 4718 observations
##   mean=0.6477321, MSE=0.2281752 
## 
## Node number 12: 9336 observations
##   mean=0.1308912, MSE=0.1137587 
## 
## Node number 13: 1115 observations
##   mean=1, MSE=0 
## 
## Node number 14: 9989 observations,    complexity param=0.01904935
##   mean=0.6178797, MSE=0.2361044 
##   left son=28 (6957 obs) right son=29 (3032 obs)
##   Primary splits:
##       Online.boarding       < 4.5      to the left,  improve=0.20547210, (0 missing)
##       Checkin.service       < 2.5      to the left,  improve=0.17629500, (0 missing)
##       Inflight.wifi.service < 4.5      to the left,  improve=0.12251280, (0 missing)
##       Seat.comfort          < 4.5      to the left,  improve=0.11016760, (0 missing)
##       Cleanliness           < 2.5      to the left,  improve=0.07822242, (0 missing)
##   Surrogate splits:
##       Inflight.wifi.service  < 4.5      to the left,  agree=0.779, adj=0.271, (0 split)
##       Ease.of.Online.booking < 4.5      to the left,  agree=0.736, adj=0.130, (0 split)
##       Cleanliness            < 4.5      to the left,  agree=0.725, adj=0.095, (0 split)
##       Seat.comfort           < 4.5      to the left,  agree=0.721, adj=0.082, (0 split)
##       Checkin.service        < 4.5      to the left,  agree=0.699, adj=0.009, (0 split)
## 
## Node number 15: 30883 observations
##   mean=0.9262054, MSE=0.06834899 
## 
## Node number 16: 32928 observations
##   mean=0.01658163, MSE=0.01630668 
## 
## Node number 17: 12861 observations,    complexity param=0.01490694
##   mean=0.1886323, MSE=0.1530502 
##   left son=34 (9015 obs) right son=35 (3846 obs)
##   Primary splits:
##       Inflight.entertainment < 3.5      to the left,  improve=0.20442150, (0 missing)
##       Leg.room.service       < 3.5      to the left,  improve=0.16640300, (0 missing)
##       On.board.service       < 3.5      to the left,  improve=0.15337020, (0 missing)
##       Inflight.service       < 3.5      to the left,  improve=0.09739190, (0 missing)
##       Baggage.handling       < 3.5      to the left,  improve=0.09394271, (0 missing)
##   Surrogate splits:
##       Cleanliness      < 3.5      to the left,  agree=0.828, adj=0.426, (0 split)
##       Food.and.drink   < 3.5      to the left,  agree=0.790, adj=0.299, (0 split)
##       Seat.comfort     < 3.5      to the left,  agree=0.784, adj=0.279, (0 split)
##       On.board.service < 3.5      to the left,  agree=0.745, adj=0.148, (0 split)
##       Inflight.service < 4.5      to the left,  agree=0.739, adj=0.126, (0 split)
## 
## Node number 28: 6957 observations,    complexity param=0.01055337
##   mean=0.4724738, MSE=0.2492423 
##   left son=56 (1852 obs) right son=57 (5105 obs)
##   Primary splits:
##       Checkin.service       < 2.5      to the left,  improve=0.15482680, (0 missing)
##       Seat.comfort          < 4.5      to the left,  improve=0.13168950, (0 missing)
##       Cleanliness           < 2.5      to the left,  improve=0.09278151, (0 missing)
##       Inflight.wifi.service < 4.5      to the left,  improve=0.07119093, (0 missing)
##       Class                 splits as  RLL,          improve=0.06046703, (0 missing)
##   Surrogate splits:
##       Age              < 68.5     to the right, agree=0.735, adj=0.003, (0 split)
##       Leg.room.service < 0.5      to the left,  agree=0.734, adj=0.001, (0 split)
## 
## Node number 29: 3032 observations
##   mean=0.9515172, MSE=0.04613226 
## 
## Node number 34: 9015 observations
##   mean=0.07310039, MSE=0.06775672 
## 
## Node number 35: 3846 observations,    complexity param=0.01490694
##   mean=0.4594384, MSE=0.2483548 
##   left son=70 (1556 obs) right son=71 (2290 obs)
##   Primary splits:
##       Customer.Type       splits as  LR,           improve=0.4831552, (0 missing)
##       Leg.room.service    < 3.5      to the left,  improve=0.3100923, (0 missing)
##       Age                 < 32.5     to the left,  improve=0.2467983, (0 missing)
##       Flight.Distance_log < 7.345042 to the left,  improve=0.2363762, (0 missing)
##       Food.and.drink      < 3.5      to the right, improve=0.2089187, (0 missing)
##   Surrogate splits:
##       Food.and.drink      < 3.5      to the right, agree=0.727, adj=0.325, (0 split)
##       On.board.service    < 3.5      to the left,  agree=0.713, adj=0.292, (0 split)
##       Cleanliness         < 3.5      to the right, agree=0.712, adj=0.289, (0 split)
##       Leg.room.service    < 3.5      to the left,  agree=0.706, adj=0.274, (0 split)
##       Flight.Distance_log < 7.24136  to the left,  agree=0.706, adj=0.273, (0 split)
## 
## Node number 56: 1852 observations
##   mean=0.1463283, MSE=0.1249163 
## 
## Node number 57: 5105 observations
##   mean=0.5907933, MSE=0.2417566 
## 
## Node number 70: 1556 observations
##   mean=0.03920308, MSE=0.0376662 
## 
## Node number 71: 2290 observations
##   mean=0.7449782, MSE=0.1899857
print(tree_model_all$variable.importance)
##              Online.boarding        Inflight.wifi.service 
##                 9.076455e+03                 7.261304e+03 
##                 Seat.comfort       Inflight.entertainment 
##                 4.223983e+03                 3.780147e+03 
##       Ease.of.Online.booking                        Class 
##                 3.608243e+03                 3.545249e+03 
##               Type.of.Travel                  Cleanliness 
##                 3.274419e+03                 5.219502e+02 
##                Customer.Type             On.board.service 
##                 4.614965e+02                 3.160414e+02 
##                          Age              Checkin.service 
##                 2.789687e+02                 2.727817e+02 
##               Food.and.drink             Inflight.service 
##                 2.700948e+02                 2.422767e+02 
##             Leg.room.service          Flight.Distance_log 
##                 1.998079e+02                 1.830583e+02 
##             Baggage.handling Arrival.Delay.in.Minutes_log 
##                 1.597525e+02                 8.514848e-02

중요도를 순서대로 5개의 변수를 확인했을 때, [“Online.boarding”, “Inflight.wifi.service”, “Seat.comfort”, “Inflight.entertainment”, “Ease.of.Online.booking”]임을 알 수 있었다.

위의 5가지 변수들만을 포함하여, 새로운 tree model을 만들기로 한다.

tree_model_selected = rpart(satisfaction ~ Online.boarding + Inflight.wifi.service + Seat.comfort + Inflight.entertainment + Ease.of.Online.booking, data = airline_train_set, method = "class")
summary(tree_model_selected)
## Call:
## rpart(formula = satisfaction ~ Online.boarding + Inflight.wifi.service + 
##     Seat.comfort + Inflight.entertainment + Ease.of.Online.booking, 
##     data = airline_train_set, method = "class")
##   n= 103594 
## 
##           CP nsplit rel error    xerror        xstd
## 1 0.51012317      0 1.0000000 1.0000000 0.003552483
## 2 0.03893356      1 0.4898768 0.4898768 0.002931653
## 3 0.03705147      2 0.4509433 0.4509433 0.002842710
## 4 0.03104885      4 0.3768403 0.3768403 0.002650023
## 5 0.01599216      5 0.3457915 0.3457915 0.002558838
## 6 0.01000000      6 0.3297993 0.3297993 0.002509133
## 
## Variable importance
##        Online.boarding  Inflight.wifi.service           Seat.comfort 
##                     30                     26                     15 
## Inflight.entertainment Ease.of.Online.booking 
##                     15                     13 
## 
## Node number 1: 103594 observations,    complexity param=0.5101232
##   predicted class=0  expected loss=0.4333938  P(node) =1
##     class counts: 58697 44897
##    probabilities: 0.567 0.433 
##   left son=2 (52271 obs) right son=3 (51323 obs)
##   Primary splits:
##       Online.boarding        < 3.5 to the left,  improve=17076.920, (0 missing)
##       Inflight.entertainment < 3.5 to the left,  improve= 8782.078, (0 missing)
##       Inflight.wifi.service  < 3.5 to the left,  improve= 8531.767, (0 missing)
##       Seat.comfort           < 3.5 to the left,  improve= 7471.493, (0 missing)
##       Ease.of.Online.booking < 3.5 to the left,  improve= 3305.320, (0 missing)
##   Surrogate splits:
##       Seat.comfort           < 3.5 to the left,  agree=0.730, adj=0.455, (0 split)
##       Inflight.wifi.service  < 3.5 to the left,  agree=0.714, adj=0.424, (0 split)
##       Ease.of.Online.booking < 3.5 to the left,  agree=0.676, adj=0.346, (0 split)
##       Inflight.entertainment < 3.5 to the left,  agree=0.659, adj=0.311, (0 split)
## 
## Node number 2: 52271 observations,    complexity param=0.03893356
##   predicted class=0  expected loss=0.1489162  P(node) =0.5045756
##     class counts: 44487  7784
##    probabilities: 0.851 0.149 
##   left son=4 (50507 obs) right son=5 (1764 obs)
##   Primary splits:
##       Inflight.wifi.service  < 0.5 to the right, improve=2616.62600, (0 missing)
##       Ease.of.Online.booking < 0.5 to the right, improve=1119.89000, (0 missing)
##       Online.boarding        < 0.5 to the right, improve= 843.53630, (0 missing)
##       Inflight.entertainment < 3.5 to the left,  improve= 543.38630, (0 missing)
##       Seat.comfort           < 4.5 to the left,  improve=  26.70475, (0 missing)
##   Surrogate splits:
##       Ease.of.Online.booking < 0.5 to the right, agree=0.971, adj=0.151, (0 split)
##       Online.boarding        < 0.5 to the right, agree=0.968, adj=0.041, (0 split)
##       Inflight.entertainment < 0.5 to the right, agree=0.966, adj=0.002, (0 split)
## 
## Node number 3: 51323 observations,    complexity param=0.03705147
##   predicted class=1  expected loss=0.2768739  P(node) =0.4954244
##     class counts: 14210 37113
##    probabilities: 0.277 0.723 
##   left son=6 (16097 obs) right son=7 (35226 obs)
##   Primary splits:
##       Inflight.entertainment < 3.5 to the left,  improve=3047.6900, (0 missing)
##       Inflight.wifi.service  < 4.5 to the left,  improve=1787.6230, (0 missing)
##       Seat.comfort           < 3.5 to the left,  improve=1535.5770, (0 missing)
##       Online.boarding        < 4.5 to the left,  improve=1527.3160, (0 missing)
##       Ease.of.Online.booking < 4.5 to the left,  improve= 580.3295, (0 missing)
##   Surrogate splits:
##       Seat.comfort           < 3.5 to the left,  agree=0.786, adj=0.318, (0 split)
##       Ease.of.Online.booking < 0.5 to the left,  agree=0.689, adj=0.007, (0 split)
##       Inflight.wifi.service  < 0.5 to the left,  agree=0.688, adj=0.006, (0 split)
## 
## Node number 4: 50507 observations,    complexity param=0.03104885
##   predicted class=0  expected loss=0.1193498  P(node) =0.4875475
##     class counts: 44479  6028
##    probabilities: 0.881 0.119 
##   left son=8 (45789 obs) right son=9 (4718 obs)
##   Primary splits:
##       Inflight.wifi.service  < 3.5 to the left,  improve=2905.861000, (0 missing)
##       Ease.of.Online.booking < 3.5 to the left,  improve= 800.324200, (0 missing)
##       Inflight.entertainment < 3.5 to the left,  improve= 544.586800, (0 missing)
##       Seat.comfort           < 4.5 to the left,  improve=  18.642340, (0 missing)
##       Online.boarding        < 2.5 to the left,  improve=   6.842733, (0 missing)
## 
## Node number 5: 1764 observations
##   predicted class=1  expected loss=0.004535147  P(node) =0.01702801
##     class counts:     8  1756
##    probabilities: 0.005 0.995 
## 
## Node number 6: 16097 observations,    complexity param=0.03705147
##   predicted class=0  expected loss=0.4682239  P(node) =0.1553854
##     class counts:  8560  7537
##    probabilities: 0.532 0.468 
##   left son=12 (13791 obs) right son=13 (2306 obs)
##   Primary splits:
##       Inflight.wifi.service  < 4.5 to the left,  improve=1519.8040, (0 missing)
##       Online.boarding        < 4.5 to the left,  improve= 794.8197, (0 missing)
##       Ease.of.Online.booking < 4.5 to the left,  improve= 614.7695, (0 missing)
##       Seat.comfort           < 4.5 to the left,  improve= 180.4196, (0 missing)
##       Inflight.entertainment < 2.5 to the left,  improve= 139.9171, (0 missing)
##   Surrogate splits:
##       Ease.of.Online.booking < 4.5 to the left,  agree=0.919, adj=0.436, (0 split)
## 
## Node number 7: 35226 observations
##   predicted class=1  expected loss=0.1603929  P(node) =0.340039
##     class counts:  5650 29576
##    probabilities: 0.160 0.840 
## 
## Node number 8: 45789 observations
##   predicted class=0  expected loss=0.06490642  P(node) =0.4420044
##     class counts: 42817  2972
##    probabilities: 0.935 0.065 
## 
## Node number 9: 4718 observations
##   predicted class=1  expected loss=0.3522679  P(node) =0.04554318
##     class counts:  1662  3056
##    probabilities: 0.352 0.648 
## 
## Node number 12: 13791 observations,    complexity param=0.01599216
##   predicted class=0  expected loss=0.3793779  P(node) =0.1331255
##     class counts:  8559  5232
##    probabilities: 0.621 0.379 
##   left son=24 (13073 obs) right son=25 (718 obs)
##   Primary splits:
##       Inflight.wifi.service  < 0.5 to the right, improve=583.4847, (0 missing)
##       Ease.of.Online.booking < 0.5 to the right, improve=389.3154, (0 missing)
##       Seat.comfort           < 4.5 to the left,  improve=232.0426, (0 missing)
##       Online.boarding        < 4.5 to the left,  improve=215.4442, (0 missing)
##       Inflight.entertainment < 2.5 to the left,  improve=160.5671, (0 missing)
##   Surrogate splits:
##       Ease.of.Online.booking < 0.5 to the right, agree=0.988, adj=0.773, (0 split)
## 
## Node number 13: 2306 observations
##   predicted class=1  expected loss=0.0004336513  P(node) =0.02225998
##     class counts:     1  2305
##    probabilities: 0.000 1.000 
## 
## Node number 24: 13073 observations
##   predicted class=0  expected loss=0.3452918  P(node) =0.1261946
##     class counts:  8559  4514
##    probabilities: 0.655 0.345 
## 
## Node number 25: 718 observations
##   predicted class=1  expected loss=0  P(node) =0.006930903
##     class counts:     0   718
##    probabilities: 0.000 1.000
train_pred_all = predict(tree_model_selected, airline_train_set, type = "class")
test_pred_all = predict(tree_model_selected, airline_test_set, type = "class")

train_conf_matrix_all = table(airline_train_set$satisfaction, train_pred_all)
test_conf_matrix_all = table(airline_test_set$satisfaction, test_pred_all)

# 훈련 오류율 계산
train_error_rate = 1 - sum(diag(train_conf_matrix_all)) / sum(train_conf_matrix_all)
test_error_rate = 1 - sum(diag(test_conf_matrix_all)) / sum(test_conf_matrix_all)

cat("train Error Rate:", train_error_rate, "\n")
## train Error Rate: 0.142933
table(train_pred_all)
## train_pred_all
##     0     1 
## 58862 44732
cat("test Error Rate:", test_error_rate, "\n")
## test Error Rate: 0.1467578
# 단말 노드의 개수 확인
num_terminal_nodes = sum(tree_model_selected$frame$var == "<leaf>")
cat("Number of Terminal Nodes:", num_terminal_nodes, "\n")
## Number of Terminal Nodes: 7

변수를 선택해서 넣은 경우의 training error rate: 14%, test error rate: 15%
상당히 낮은 error rate을 보인다.
매우 잘 동작하는 것을 확인할 수 있다.

rpart.plot(tree_model_selected)

print(tree_model_selected$variable.importance)
##        Online.boarding  Inflight.wifi.service           Seat.comfort 
##              17183.720              14878.110               8731.368 
## Inflight.entertainment Ease.of.Online.booking 
##               8368.055               7440.170

root node : Online.boarding(기준 : 4)
온라인 탑승 수속 만족도가 4보다 작다면 왼쪽, 크다면 오른쪽으로 데이터를 분할한다.

satisfaction의 예측에 있어서 해당 모델에서 Online.boarding이 가장 중요한 변수로 사용되었다. 만족도에 있어서 Online.boarding(온라인 탑승 수속 만족도)이 중요한 요소라고 할 수 있다.

#tree_model_selected에 대한 confusion matrix
test_conf_matrix_all
##    test_pred_all
##         0     1
##   0 12671  1857
##   1  1943  9422
TP = 9422
TN = 12671
FP = 1857
FN = 1943

accuracy = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
specificity = TN / (TN + FP)

f1_score = 2 * (precision * recall) / (precision + recall)

print(paste("Accuracy:", accuracy))
## [1] "Accuracy: 0.853242189008612"
print(paste("Precision:", precision))
## [1] "Precision: 0.835357744480894"
print(paste("Recall:", recall))
## [1] "Recall: 0.829036515618126"
print(paste("Specificity:", specificity))
## [1] "Specificity: 0.872177863436123"
print(paste("F1 Score:", f1_score))
## [1] "F1 Score: 0.832185126302773"

satisfaction을 예측하는 tree model의 경우, f1 score가 0.83, accuracy가 0.85로 이전의 satisfaction을 예측하는 모델 중에서 가장 좋은 성능을 보이는 것을 확인할 수 있다.